x

75% of people globally want unbiased news (Pew Research) but are concerned they're not getting that.

Get balanced coverage with Symby. See how stories are covered across diverse outlets and spot when one side covers a story the other skips.

PODCAST

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage:

technology

All Episodes

22:28

Transforming the Airflow UI for Cloudera’s Users with...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/07/10

Contributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj, Software Engineer II at Cloudera, shares how his contributions helped shape Airflow 3.0, including an intuitive drag-and-drop DAG editor and a new REST API endpoint for managing XComs.Key Takeaways:(02:30) Day-to-day responsibilities building

en

19:34

Streamlining Thousands of Data Pipelines at Lyft with...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/07/07

Managing data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows.In this episode, we speak with Yunhao Qing, Software Engineer at Lyft, about building a governed data-engineering platform powered by Airflow that balances flexibility, standardization and scale.Key Takeaways:(03:17) Supporting internal teams with a

en

22:19

Transforming Customer Education in Data Engineering...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/06/26

Understanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone.In this episode, Marc Lamberti, Head of Customer Education at Astronomer, joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners demystify Airflow.Key Takeaways:(02:36) Early exposure to Airflow while addressing inefficiencies in data

en

30:09

Embracing Data Mesh and SQL Sensors for Scalable...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/06/19

The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.Key Takeaways:(02:17) Defining responsibilities within data architecture teams.(04:15) Consolidating multiple orchestrators into a single

en

23:21

The AI-Ready Pipeline: Reimagining Airflow at Veyer®...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/06/12

Innovation in orchestration is redefining how engineers approach both traditional ETL pipelines and emerging AI workloads. Understanding how to harness Airflow’s flexibility and observability is essential for teams navigating today’s evolving data landscape.In this episode, Anu Pabla, Principal Engineer at The ODP Corporation, joins us to discuss her journey from legacy orchestration patterns to AI-native pipelines and why she sees Airflow as the future of AI workload orchestration.Key

en

24:44

Streamlining AI and ML Operations at IBM with BJ...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/06/05

The orchestration layer is foundational to building robust AI- and ML-powered data pipelines, especially in complex hybrid enterprise environments. IBM’s partnership with Astronomer reflects a strategic alignment to simplify and scale Airflow-based workflows across industries.In this episode, we’re joined by IBM’s Senior Product Manager, BJ Adesoji, and Chief Marketing Officer, Ryan Yackel. We discuss how IBM customers are using Airflow in production, the challenges they face at scale and what the

en

31:02

Inside the Custom Framework for Managing Airflow Code...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/05/29

Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He

en

22:13

Modernizing Legacy Data Systems With Airflow at...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/05/22

Legacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures.Adonis Castillo Cordero, Senior Automation Manager at Procter & Gamble, joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow.Key Takeaways:(03:13) Integrating

en

38:54

Building an End-to-End Data Observability System at...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/05/15

Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He

en

30:28

Why Developer Experience Shapes Data Pipeline...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/05/08

Creating consistency across data pipelines is critical for scaling engineering teams and ensuring long-term maintainability.In this episode, Snir Israeli, Senior Data Engineer at Next Insurance, shares how enforcing coding standards and investing in developer experience transformed their approach to data engineering. He explains how implementing automated code checks, clear documentation practices and a scoring system helped drive alignment across teams, improve collaboration and reduce technical

en

22:49

Data Quality and Observability at Tekmetric with Ipsa...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/05/01

Airflow’s adaptability is driving Tekmetric’s ability to unify complex data workflows, deliver accurate insights and support both internal operations and customer-facing services — all within a rapidly growing startup environment.In this episode, Ipsa Trivedi, Lead Data Engineer at Tekmetric, shares how her team is standardizing pipelines while supporting unique customer needs. She explains how Airflow enables end-to-end data services, simplifies orchestration across varied sources and supports

en

27:28

Introducing Apache Airflow® 3 with Vikram Koka and...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/04/24

The Airflow 3.0 release marks a significant leap forward in modern data orchestration, introducing architectural upgrades that improve scalability, flexibility and long-term maintainability.In this episode, we welcome Vikram Koka, Chief Strategy Officer at Astronomer, and Jed Cunningham, Principal Software Engineer at Astronomer, to discuss the architectural foundations, new features and future implications of this milestone release. They unpack the rationale behind DAG versioning and task execution

en

25:14

Airflow in Action: Powering Instacart's Complex...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/04/17

The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of

en

27:42

From ETL to Airflow: Transforming Data Engineering at...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/04/10

Data orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference.In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital, joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows.Key Takeaways:(01:45) Early challenges in data orchestration before implementing Airflow.(02:42)

en

23:26

A Deep Dive Into the 2025 State of Airflow Survey...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/04/03

The 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community.In this episode, Tamara Fingerlin, Developer Advocate at Astronomer, walks us through her process of analyzing survey data, key trends from the report and what to expect from Airflow 3.0.Key

en

26:15

Airflow’s Role in the Rise of DataOps with Andy Byron

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/03/27

The orchestration layer is evolving into a critical component of the modern data stack. Understanding its role in DataOps is key to optimizing workflows, improving reliability and reducing complexity.In this episode, Andy Byron, CEO at Astronomer, discusses the rapid growth of Apache Airflow, the increasing importance of orchestration and how Astronomer is shaping the future of DataOps.Key Takeaways:(01:54) Orchestration is central to modern data workflows.(03:16) Airflow 3.0 will enhance usability

en

28:27

The Software Risk That Affects Everyone and How To...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/03/20

The security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively.In this episode, we sit down with Michael Winser, Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation, and Jarek Potiuk, Member of the Security Committee at the Apache Software Foundation, to discuss the challenges of securing Airflow’s dependencies, the evolving

en

36:46

Building Scalable ML Infrastructure at Outerbounds...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/03/13

Machine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal, Co-Founder and CTO at Outerbounds, about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to simplify data science.Key Takeaways:(02:02) Savin spent years building AI and ML infrastructure,

en

27:40

Customizing Airflow for Complex Data Environments at...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/03/06

Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience.Key

en

17:54

Harnessing Airflow for Data-Driven Policy Research at...

The Data Flowcast: Mastering Apache Airflow ® for... ·

2025/02/27

Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets.Key Takeaways:(02:04) CSET provides data-driven analysis

en

84 results

Similar Podcasts

Lead With AI

Tamara Nall

Linux in the Ham Shack

Black Sparrow Media

Data Engineering Podcast

Tobias Macey

Eye On A.I.

Craig S. Smith

Reflections

Livewire Labs