PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Byte Sized Breakthroughs
19/07/2024
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Listen "PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel"

Descargar episodio Ver en sitio original

Episode Synopsis




FSDP addresses memory capacity challenges by sharding parameters across devices, employs communication optimizations to enhance efficiency, includes a rate limiter feature to control memory impact, offers user-friendly APIs for easy integration, achieved promising results on large models, enables broader applications in various domains, faces challenges in mathematical equivalence and handling shared parameters, and has potential research directions in adaptive sharding strategies, new communication primitives, and combining with other parallelism paradigms.

Read full paper: https://arxiv.org/abs/2304.11277

Tags: Systems and Performance, Deep Learning, Machine Learning

More episodes of the podcast Byte Sized Breakthroughs

TransAct Transformer-based Realtime User Action Model for Recommendation at Pinterest 08/07/2024
Zero Bubble Pipeline Parallelism 08/07/2024
The limits to learning a diffusion model 08/07/2024
A Better Match for Drivers and Riders Reinforcement Learning at Lyft 08/07/2024
AutoEmb Automated Embedding Dimensionality Searchg in Streaming Recommendations 08/07/2024
NeuralProphet Explainable Forecasting at Scale 08/07/2024
No-Transaction Band Network A Neural Network Architecture for Efficient Deep Hedging 08/07/2024
ZeRO Memory Optimizations: Toward Training Trillion Parameter Models 08/07/2024
DriveVLM: Vision-Language Models for Autonomous Driving in Urban Environments 18/07/2024
Robustness Evaluation of HD Map Constructors under Sensor Corruptions for Autonomous Driving 18/07/2024
© 2003-2026 ZARZA | AHEAD OF OUR TIME
privacy policy
In God we trust In God we trust
PreSales Questions? Click Here
or call our toll-free line at +1-888-5002068 Miami, Florida, USA