Reliability, in One Place

53How SLA, timeout, and retry mechanisms fit together#

How it works:

• Two independent SLA clocks: DAG-level SLA (feature 7) measures elapsed time from the run's actual start; task-level SLA (feature 17) measures elapsed time from the run's logical date instead. A run that queues for a while before actually starting has already consumed part of its tasks' SLA budgets by the time it begins, even though the DAG-level clock hasn't started yet.
• Three independent timeout layers: task-level execution_timeout (feature 16), DAG-level dagrun_timeout_seconds (feature 6), and a global abandoned-run safety net that only fires when a run has genuinely gone quiet (no task actively heartbeating) — a healthy, slow DAG is never killed by that global fallback alone.
• Retry backoff (feature 13) applies consistently to every retryable failure, whatever caused it — including a task recovered after its worker process itself was restarted, not just an ordinary in-process failure. You can rely on your configured fixed or exponential backoff curve regardless of the underlying cause.
• When a run is force-failed by a timeout, its still-running/scheduled/ queued/up_for_retry tasks are swept into failed along with it. Tasks that are deferred (feature 28) or up_for_reschedule (feature 51) at that moment are not automatically swept — if your run's timeout might fire while a deferred/reschedule-mode task is still in flight, plan to check on that task manually afterward.
• All hard failures (retries exhausted, timeout, or an infrastructure-level recovery) converge on the same dead-letter queue, which is a reasonable single place to review any of these failure modes.

Example:

with DAG(
    dag_id="nightly_etl",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    dagrun_timeout_seconds=3600 * 4,   # explicit strict run-level timeout
    expected_duration_seconds=3600 * 2,  # DAG-level SLA: flagged (not failed) past 2h
) as dag:

    process = PythonOperator(
        task_id="process",
        python_callable=lambda: None,
        execution_timeout=1800,   # task-level cap, independent of the run-level one
        sla=900,                  # task-level SLA, clocked from the run's logical date
        retries=3,
        retry_exponential_backoff=True,
        retry_delay_seconds=30,
        max_retry_delay_seconds=600,
    )