From Data Mesh to Lake House: Revolutionizing Metadata with Lakekeeper

21/03/2025 57 min Episodio 16
From Data Mesh to Lake House: Revolutionizing Metadata with Lakekeeper

Listen "From Data Mesh to Lake House: Revolutionizing Metadata with Lakekeeper"

Episode Synopsis


SummaryIn this episode, Viktor Kessler shares his journey and insights from his extensive experience in data management—from building risk management systems and data warehouses to working as a solutions architect at MongoDB and Dremio, and now co-founding a startup.Initially exploring data mesh concepts, Viktor explains how real-world challenges—such as the disconnect between technical data models and business needs, inconsistent definitions across departments, and the difficulty in managing actionable metadata—led him and his co-founder to pivot toward building a lake house solution. His startup is developing Lakekeeper, an open source REST catalog for Apache Iceberg, which aims to bridge the gap between decentralized data production and centralized metadata management. The conversation also delves into the evolution of data catalogs, the necessity for self-service analytics, and how creating consumption-ready data products can transform data functions from cost centers into profit centers. Finally, Viktor outlines ways for interested listeners to get involved with the Lakekeeper community through GitHub, upcoming meetups, and a dedicated Discord channel.Chapters00:00 Introduction to Viktor Kessler and His Journey04:57 Transitioning from Data Mesh to Lake House09:15 Understanding Data Mesh: Pain Points and Solutions13:47 The Role of Metadata in Data Management18:16 The Evolution of Catalogs and Metadata Management28:14 Stabilizing the Consumption Pipeline31:18 Centralizing Metadata for Decentralized Organizations37:09 Bridging the Gap: Technical and Business Perspectives43:17 Rethinking Data Products and Consumption50:45 Finding Balance: Control and Flexibility in Data Management

More episodes of the podcast Tech on the Rocks