Zero Bubble Pipeline Parallelism

Byte Sized Breakthroughs
08/07/2024
Zero Bubble Pipeline Parallelism

Listen "Zero Bubble Pipeline Parallelism"

Descargar episodio Ver en sitio original

Episode Synopsis




Core idea is think about backward pass into two flows, one to compute grad wrt to parameters, and one to compute grad wrt to output of last layer,
schedule so that you are always working instead of waiting (bubble).

Read full paper: https://arxiv.org/abs/2401.10241

Tags: Systems and Performance, Deep Learning, Machine Learning

More episodes of the podcast Byte Sized Breakthroughs

TransAct Transformer-based Realtime User Action Model for Recommendation at Pinterest 08/07/2024
The limits to learning a diffusion model 08/07/2024
A Better Match for Drivers and Riders Reinforcement Learning at Lyft 08/07/2024
AutoEmb Automated Embedding Dimensionality Searchg in Streaming Recommendations 08/07/2024
NeuralProphet Explainable Forecasting at Scale 08/07/2024
No-Transaction Band Network A Neural Network Architecture for Efficient Deep Hedging 08/07/2024
ZeRO Memory Optimizations: Toward Training Trillion Parameter Models 08/07/2024
DriveVLM: Vision-Language Models for Autonomous Driving in Urban Environments 18/07/2024
Robustness Evaluation of HD Map Constructors under Sensor Corruptions for Autonomous Driving 18/07/2024
RT-DETR: Real-Time Object Detection with Transformer 18/07/2024
© 2003-2026 ZARZA | AHEAD OF OUR TIME
privacy policy
In God we trust In God we trust
PreSales Questions? Click Here
or call our toll-free line at +1-888-5002068 Miami, Florida, USA