In this episode, Kostas and Nitay are joined by Amey Chaugule and Matt Green, co-founders of Denormalized. They delve into how Denormalized is building an embedded stream processing engine—think “DuckDB for streaming”—to simplify real-time data workloads. Drawing from their extensive backgrounds at companies like Uber, Lyft, Stripe, and Coinbase. Amey and Matt discuss the challenges of existing stream processing systems like Spark, Flink, and Kafka. They explain how their approach leverages Apache DataFusion, to create a single-node solution that reduces the complexities inherent in distributed systems.The conversation explores topics such as developer experience, fault tolerance, state management, and the future of stream processing interfaces. Whether you’re a data engineer, application developer, or simply interested in the evolution of real-time data infrastructure, this episode offers valuable insights into making stream processing more accessible and efficient.Contacts & LinksAmey ChauguleMatt GreenDenormalizedDenormalized Github RepoChapters00:00 Introduction and Background12:03 Building an Embedded Stream Processing Engine18:39 The Need for Stream Processing in the Current Landscape22:45 Interfaces for Interacting with Stream Processing Systems26:58 The Target Persona for Stream Processing Systems31:23 Simplifying Stream Processing Workloads and State Management34:50 State and Buffer Management37:03 Distributed Computing vs. Single-Node Systems42:28 Cost Savings with Single-Node Systems47:04 The Power and Extensibility of Data Fusion55:26 Integrating Data Store with Data Fusion57:02 The Future of Streaming Systems01:00:18 intro-outro-fade.mp3Click here to view the episode transcript.

Ver todos los episodios