An interview with Greg Harris and Ivan Yurchenko, part of the authors of the new KIP-1150 that introduces diskless topics to Apache Kafka.Diskless topics are a special kind of topic that write directly to an object store. This outsources the data replication and durability guarantees to the store (e.g., S3) and avoids the costly inter-zone data transfer fees that clouds charge you for. The result is a 80%+ reduced Kafka bill and a simpler architecture where brokers are stateless (or have less state) – making them significantly easier to maintain operationally.In this podcast, we dive deep into the technical details and tradeoffs in this complex KIP.⏱️ TIMELINE 04:00 – Why Diskless? 09:17 – Architecture Walkthrough: How Does It Work 18:50 – The Batch Coordinator 22:00 – The Different Batch Coordinator Implementations 28:00 – The Inkless Fork (And Why) 39:40 – Topic-Based Batch Coordinator 45:40 – Bottlenecks in the Batch Coordinator? 47:48 – The Read Path 54:40 – Caching 01:03:00 – Shared Log Segment Merging (AKA Object Compaction) 01:19:06 – Latency 01:26:30 – Practical Real-World Latency Requirements 01:39:00 – The Power of Open Source 01:47:30 – Cost Breakdown 01:58:00 – Ops and Engineering Time Savings 02:06:30 – Client Bootstrapping, Leaderless Partitions, One Broker Connection per Client 02:11:20 – Multi-Tenant Kafka Future 02:14:05 – Iceberg, Parquet, and First-Class Schema Support 02:23:10 – S3 ExpressIf you found this episode interesting, please give it a like to signal to the algorithm that it's good. It takes 2 seconds to do, and it took us 5 hours to produce.💬 HOW TO GET INVOLVEDWe recommend participating in the Apache Kafka mailing list.Beyond that, you can find Greg and Ivan here: Ivan Yurchenko – X: @ivan0yu Greg Harris – X: @gharris1727⚠️ DISCLAIMERSAll views expressed in this podcast are personal opinions of the participants. They do not represent the views or positions of Aiven, the employer of Ivan and Greg, or any other entity.Apache®, Apache Kafka®, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Ver todos los episodios