Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 - Episode #67

30/06/2025 22 min Episodio 67
Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 - Episode #67

Listen "Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 - Episode #67"

Episode Synopsis

Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Orchestrating Dynamic Data Flows.

The Dynamic Data Flow Orchestration pattern dynamically generates and self-heals data flow manifests (DAGs) at runtime from a central context database, enabling adaptive, cost-effective, and low-latency data processing primarily for micro-batching workloads

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.
 
If you want a copy of the pattern template head over to:
https://agiledata.substack.com/i/167144707/pattern-name


Discover more  AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns
 
 
If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.
 
Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict | 



Buy the Green Book now!



 
Simply Magical Data Ways of Working

More episodes of the podcast Agile Data N’ Info