ZeRO Memory Optimizations: Toward Training Trillion Parameter Models

Byte Sized Breakthroughs
08/07/2024
ZeRO Memory Optimizations: Toward Training Trillion Parameter Models

Listen "ZeRO Memory Optimizations: Toward Training Trillion Parameter Models"

Descargar episodio Ver en sitio original

Episode Synopsis




The paper introduces ZeRO, a novel approach to optimize memory usage when training massive language models. ZeRO-DP and ZeRO-R components effectively reduce memory redundancy and allow for training models with up to 170 billion parameters efficiently. The technique shows superlinear scalability, user-friendly implementation, and has the potential to democratize large model training in AI research.

Read full paper: https://arxiv.org/abs/1910.02054

Tags: Systems and Performance, Deep Learning, Natural Language Processing

More episodes of the podcast Byte Sized Breakthroughs

TransAct Transformer-based Realtime User Action Model for Recommendation at Pinterest 08/07/2024
Zero Bubble Pipeline Parallelism 08/07/2024
The limits to learning a diffusion model 08/07/2024
A Better Match for Drivers and Riders Reinforcement Learning at Lyft 08/07/2024
AutoEmb Automated Embedding Dimensionality Searchg in Streaming Recommendations 08/07/2024
NeuralProphet Explainable Forecasting at Scale 08/07/2024
No-Transaction Band Network A Neural Network Architecture for Efficient Deep Hedging 08/07/2024
DriveVLM: Vision-Language Models for Autonomous Driving in Urban Environments 18/07/2024
Robustness Evaluation of HD Map Constructors under Sensor Corruptions for Autonomous Driving 18/07/2024
RT-DETR: Real-Time Object Detection with Transformer 18/07/2024
© 2003-2026 ZARZA | AHEAD OF OUR TIME
privacy policy
In God we trust In God we trust
PreSales Questions? Click Here
or call our toll-free line at +1-888-5002068 Miami, Florida, USA