Amazon S3 Deep Dive: Scale, Decorrelation, and Velocity

23/12/2024 25 min

Listen "Amazon S3 Deep Dive: Scale, Decorrelation, and Velocity"

Episode Synopsis

This transcript from an AWS re:Invent 2024 session details Amazon S3's architecture and engineering principles. Two senior engineers explain how S3's massive scale enables efficient data management, utilising techniques like shuffle sharding to distribute workloads across millions of drives. They discuss the physics of data storage, showcasing how S3's design improves performance and reliability by mitigating the impact of hardware limitations and individual workload bursts. Erasure coding is highlighted as a key technology that ensures data durability and facilitates faster software deployment. Finally, the presentation emphasises how S3's fault-tolerant design, built on principles of decorrelation, ultimately benefits customers by providing a consistently high-performing and reliable service.