Design every task to be idempotent
The single most important property a production DAG can have is idempotency: running a task twice for the same execution date produces the same result as running it once. Without this, a retry after a failure — the exact scenario Airflow exists to handle — can corrupt data by double-inserting rows or double-charging an API.
In practice this means writing tasks that delete-and-replace a partition rather than append to it, use upserts rather than blind inserts, and check for existing output before recreating it. Idempotency is a design decision made when a task is written, not something fixed afterward by adding retries.
Get task granularity right
Too coarse and a single failure forces a rerun of an hour of work to fix a one-minute problem; too fine and the DAG becomes hundreds of tasks that are slow to render in the UI and expensive to schedule. A reasonable default is one task per logical, independently retryable unit of work — one table load, one file processed, one API endpoint called — rather than one giant task per pipeline or one microscopic task per row.
Keep the top level of a DAG file cheap
The Scheduler parses every DAG file repeatedly, on a short interval, to detect changes and pick up new runs. Any expensive operation placed at the top level of a DAG file — a database query, an API call, a large file read — runs on every single parse, not just when the DAG executes. This is one of the most common causes of a slow, unresponsive Airflow instance, and it is invisible until DAG counts grow.
- Move expensive logic inside task functions (PythonOperator callables, or the body of a custom operator), which only run at execution time.
- Avoid network calls, file system scans or heavy imports at the top level of a DAG file.
- Keep DAG generation (if generating DAGs dynamically from configuration) as lightweight as possible — it still runs on every scheduler parse.
Plan for backfills from day one
A backfill — rerunning a DAG across a historical date range, typically after a bug fix or a late-arriving requirement — is a routine operational need, not an edge case. DAGs that assume they only ever run for the current date break the first time someone needs to backfill three months of history.
Use the logical execution date Airflow provides to every task rather than calling a real-time clock function inside task logic, and confirm that downstream systems (data warehouses, APIs with rate limits) can tolerate a burst of historical runs before triggering a large backfill.
Use sensors carefully — they are a common cost surprise
A sensor is a task that waits for a condition — a file to appear, an upstream DAG to finish, a database row to exist — before allowing the DAG to continue. Sensors are useful, but a sensor configured to poll every few seconds for hours occupies a worker slot the entire time it waits, which is a common and avoidable cause of a stalled or expensive Airflow deployment.
- Prefer deferrable (reschedule-mode) sensors, which release the worker slot between checks instead of holding it for the entire wait.
- Set a sensible poke interval and timeout rather than the tightest possible polling frequency.
- Where possible, replace a polling sensor with an event-driven trigger (a message queue notification, a dataset-aware schedule) that starts the DAG only when the condition is actually met.
Build in observability, not just retries
- Alert on task failure and on SLA misses, not only on total DAG failure — a task that silently retries three times before succeeding still deserves visibility.
- Log enough context in each task (row counts processed, source and destination identifiers) that a failure can be diagnosed from the log alone.
- Track DAG run duration over time; a pipeline that has quietly doubled in run time is an early warning of a data volume or performance problem.
- Use Airflow tags and clear DAG and task naming conventions so on-call engineers can find the right pipeline quickly at 2am.
Mistakes that turn a DAG into a support burden
- Hardcoding dates or environment-specific values instead of using Airflow variables, connections and the execution-date context.
- One giant DAG covering an entire business domain, instead of smaller DAGs connected by dataset-aware scheduling — the giant DAG becomes a single point of failure and a slow one to reason about.
- No retries configured, or retries configured with no delay, which turns a transient network blip into a hard failure or a thundering-herd retry storm.
- Treating the Airflow UI as documentation instead of writing DAG and task descriptions that explain intent, not just structure.
- Skipping tests entirely because a DAG is code that seems to just run — DAG-level unit tests (validating structure, dependencies and no import errors) catch a large share of production incidents before deploy.
Frequently asked questions
Running the task twice for the same execution date produces the same end state as running it once — no duplicated rows, no double-charged API calls. It is the property that makes retries and backfills safe rather than dangerous.
There is no fixed number; the guide is independence. Each task should represent one unit of work that can fail and retry on its own without needing to redo unrelated work. Dozens of tasks in a DAG is normal; hundreds is usually a sign the DAG should be split.
The most common cause is expensive code at the top level of DAG files, which the scheduler re-executes on every parse cycle regardless of whether the DAG is currently running. Move expensive logic inside task callables.
For new DAGs written primarily in Python, the TaskFlow API generally produces cleaner, more maintainable code. Classic operators remain the right choice for many existing integrations and for pipelines built primarily around specialised operators rather than Python functions.
A deferrable sensor is usually the right tool: it waits for the file without occupying a worker slot the whole time, and it integrates cleanly with the rest of the DAG dependency graph.
Changing task structure (adding, removing or reordering tasks) can affect in-flight and historical runs, so structural changes should go through the same review and testing process as any production code change, and any semantic change to what a task does should consider whether a backfill is needed.
How we can help
Production-grade pipelines with tests, lineage and monitoring built in.
Legacy Modernisation & BI MigrationModernising brittle scheduled jobs into maintainable, observable pipelines.