What Airflow actually is
Apache Airflow is an open-source platform for authoring, scheduling and monitoring workflows. A workflow in Airflow is written as Python code that describes a set of tasks and the order they must run in — not as a drag-and-drop diagram, which is the detail that most distinguishes it from older ETL scheduling tools.
Airflow does not move or transform data itself. It orchestrates: it decides when each task runs, tracks whether it succeeded, retries it if it failed, and gives visibility into the whole pipeline through a web interface. The actual work — a SQL query, a Spark job, a Python script, an API call — is performed by the task itself or by a system Airflow triggers.
What a DAG is, without the jargon
DAG stands for Directed Acyclic Graph. Stripped of the mathematics, it means three things in practice:
- Directed — each task has a defined order relative to others. Task B runs after Task A, not the other way round.
- Acyclic — the workflow cannot loop back on itself. Once a task has run, the pipeline moves forward; it cannot circle back to an earlier task in the same run.
- Graph — the workflow is a network of tasks and dependencies, not necessarily a single straight line. Several tasks can run in parallel and then converge into one.
What a DAG looks like in practice
A DAG file is ordinary Python. It defines a schedule (for example, once a day at 2am), a set of tasks (an extraction step, a transformation step, a load step, a validation step), and the dependencies between them — commonly written with the >> operator to mean run this, then that.
Because a DAG is code, it goes through the same discipline as application code: version control, code review, and automated tests. This is a meaningful advantage over pipelines built inside a proprietary drag-and-drop tool, where dependency logic is often locked inside a vendor file format that is hard to diff or review.
How Airflow actually runs a DAG
Airflow has a small number of moving parts that matter operationally:
- The Scheduler decides when each DAG should run and hands its tasks off for execution, based on the schedule and any dependencies.
- The Executor determines how tasks actually run — locally, across a Celery worker pool, or on Kubernetes, spinning up a pod per task.
- The Metadata Database stores the state of every DAG run and task instance, which is what powers retries, history and the web UI.
- The Webserver renders the UI: DAG graphs, run history, logs per task, and manual controls to trigger, pause or clear a run.
- A task instance is one specific run of one task on one schedule interval — this is the unit Airflow retries and reports status on.
Why data teams reach for Airflow specifically
Cron could technically schedule most of what Airflow schedules. What Airflow adds is everything cron does not: dependency management between tasks, automatic retries with backoff, alerting on failure, a visual history of every run, and the ability to backfill a date range if a pipeline needed to be re-run after a bug fix.
It has also become something close to a lingua franca for data engineering — most modern data stacks (dbt, Spark, cloud data warehouses, ML training jobs) have an Airflow provider or integration already built, so it functions as the connective layer between otherwise separate tools rather than one more system a team has to build integrations for.
When Airflow is not the right choice
- Very low latency or streaming requirements — Airflow is a batch scheduler at heart; for sub-minute streaming, a dedicated stream processing tool fits better.
- A single, simple scheduled script with no dependencies — a cron job may be genuinely sufficient, and running Airflow purely for one task adds operational overhead for no benefit.
- Teams with no Python or DevOps capacity to operate it — Airflow is a system that needs monitoring and occasional maintenance like any other production service.
- Extremely dynamic, per-event workflows generated at high volume — Airflow DAGs are best suited to a relatively stable, predictable set of workflows rather than one dynamically created per event.
Managed Airflow or self-hosted
All three major clouds now offer managed Airflow (Amazon MWAA, Google Cloud Composer, and Astronomer on Azure or any cloud), which removes the operational burden of running the scheduler, database and webserver. For most teams below a certain scale, managed Airflow is the sensible default — self-hosting is justified mainly by very specific customisation needs or cost sensitivity at large scale.
Frequently asked questions
Not directly. Airflow orchestrates and schedules workflows; the extraction, transformation and loading logic typically lives in the tasks it triggers (a dbt run, a Spark job, a Python script), rather than inside Airflow itself.
Yes, at least basic Python. DAGs are authored as Python files, though the tasks inside a DAG frequently call out to SQL, Spark or shell commands rather than doing heavy computation in Python itself.
Airflow marks that task instance as failed, can retry it automatically according to a configured policy, and downstream tasks that depend on it will not run until it succeeds or the failure is resolved manually.
Yes, through sensors or dataset-aware scheduling that let one DAG wait for another to complete or for a specific dataset to be updated, rather than only running on a fixed clock schedule.
Not as its primary use case. Airflow excels at scheduled, batch-oriented workflows. For genuinely real-time streaming, a dedicated stream processing framework is generally the better fit, sometimes triggered or monitored by Airflow at a higher level.
Most teams do not outgrow Airflow itself so much as its self-hosted operation, which is why managed offerings exist. Alternatives such as Dagster and Prefect solve similar problems with different programming models, but migrating away from an established Airflow estate is a significant undertaking best justified by a specific, concrete limitation.
How we can help
ELT pipelines built and orchestrated with tests, lineage and monitoring.
Solution Design & ArchitectureChoosing the right orchestration and data tooling for your platform.