Listen "185: Workflow Orchestrators"
Episode Synopsis
Intro topic: Asymmetric ReturnsNews/Links:NanoChat by Andrej Karpathyhttps://github.com/karpathy/nanochatPydantic AIhttps://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/1000th Starlink this yearhttps://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/ChatGPT Apps SDKhttps://openai.com/index/introducing-apps-in-chatgpt/Book of the ShowPatrickThe Will of the Many by James Islingtonhttps://amzn.to/43IfU8QJasonInterview with DHH (Founder of Ruby on Rails)https://www.youtube.com/watch?v=vagyIcmIGOQPatreon Plug https://www.patreon.com/programmingthrowdown?ty=hTool of the ShowPatrickFactoriohttps://www.factorio.com/ Jasonnip.io Topic: Workflow OrchestratorsWhyBatch jobs (embarrassingly parallel)Long-running tasks (e.g. transcoding video)Checkpointing/resumingHowMessage QueuesContainerizationWorker Pools & AutoscalingHistory & BackfillSteps to run workflows:Containerize the workflow definition and send to the cloudContainerize all the individual tasksSubmit job(s)ExamplesAirflowLegacy but dominantDagsterGreat UX for python developersTemporal: https://temporal.io/The new hotnessRayLow-level but very powerfulKubeflowDesigned for ML workflows, integrated dashboard
★ Support this podcast on Patreon ★
More episodes of the podcast Programming Throwdown
184: Asynchronous Programming
23/09/2025
183: Landing a Software Job in 2025
31/07/2025
182: AI Assisted Coding
30/06/2025
181: Memory Management
12/05/2025
180: Reinforcement Learning
17/03/2025
179: Project Planning
03/02/2025
178: Working from Home
03/12/2024
177: Vector Databases
04/11/2024
176: MLOps at SwampUp
24/09/2024
175: Resume Writing
16/08/2024
ZARZA We are Zarza, the prestigious firm behind major projects in information technology.