Why I stopped using Airflow sensors

Wait for a file in a bucket. Wait for a partition in a warehouse table. Wait for that other DAG's task to finish. A sensor is the obvious way to say "this depends on that", and for a long time it was the only way Airflow gave you. Here's what changed my mind.

What a sensor costs

In the default poke mode a sensor is a task that runs for its entire wait: it holds a worker slot — on Celery or Kubernetes, a whole process — to sleep and check every minute. Twenty DAGs waiting on twenty upstream tables at 06:00 is twenty slots gone, and those are the slots the actual work needed. mode="reschedule" fixes that by releasing the slot between pokes, at the cost of a task-instance state flip every interval, which adds noise to the scheduler and the UI, and a poke_interval you now have to tune per sensor.

The subtler cost is what a sensor hides. A sensor with timeout=6*3600 that's waiting on an upstream failure shows in the UI as a task that has been running for six hours. Nothing looks wrong. Then at noon it fails with AirflowSensorTimeout, which tells you nothing about what went wrong or where. Every time one of these paged me, the real failure was hours old and in a different DAG.

ExternalTaskSensor in particular

ExternalTaskSensor waits for a task in another DAG, matched by logical date. That works exactly as long as both DAGs have the same schedule. The moment upstream moves from @daily to 0 */6 * * *, or someone backfills upstream with a different interval, or downstream gets triggered by hand, the dates stop lining up and the sensor waits forever for a run that will never exist. The fix is execution_date_fn, a function that works out which upstream run to look for — and now the coupling between two DAGs' schedules lives in a lambda in the downstream file. I've written that lambda a dozen times and got it wrong at least three.

What replaced them

1. Datasets, for DAG-to-DAG dependencies

Since 2.4 a task can declare outlets=[Dataset(...)], and another DAG can be scheduled on that dataset. Downstream runs when the upstream task succeeds. No sensor, no date matching, no slot.

from datetime import datetime
from airflow import Dataset
from airflow.decorators import dag, task

orders_daily = Dataset("warehouse://analytics/orders_daily")

@dag(schedule="0 5 * * *", start_date=datetime(2025, 1, 1), catchup=False)
def build_orders():
    @task(outlets=[orders_daily])
    def load():
        ...
    load()

@dag(schedule=[orders_daily], start_date=datetime(2025, 1, 1), catchup=False)
def report_on_orders():
    @task
    def report():
        ...
    report()

build_orders()
report_on_orders()

Two things I hit. A dataset event doesn't say which partition was produced, so downstream has to work it out; I pass it in the event's extra (2.9 and later) or, honestly, just recompute "yesterday". And with several datasets in a schedule, downstream fires once all of them have updated since its last run, which is what you want for a fan-in and surprising the first time one upstream runs twice.

2. Fail fast and retry, for "is the data there yet"

If a task needs a partition that may not exist yet, check for it inside the task and raise if it's missing, with retries=12 and a ten-minute retry_delay. That's a two-hour sensor, except it holds no slot while waiting, it's visible in the UI as up_for_retry with a count, and when it finally gives up you get your exception with your message rather than a timeout.

from datetime import timedelta
from airflow.exceptions import AirflowException

@task(retries=12, retry_delay=timedelta(minutes=10))
def load_partition(ds=None):
    if not warehouse.partition_exists("raw.events", ds):
        raise AirflowException(f"raw.events/{ds} has not landed yet")
    ...

This is what most of my sensors became. It's slightly heretical — retries are meant for transient failures — but "the data isn't there yet" is a transient failure, and the retry machinery is the most robust waiting mechanism Airflow has.

3. Deferrable operators, when you genuinely must wait on the outside world

Since 2.2 a sensor can defer itself to the triggerer, one async process that waits on hundreds of things in a single event loop. S3KeySensor(deferrable=True) and its relatives cost no worker slot. I keep these for the two or three cases where an external party drops a file at an unpredictable hour and there is no way to make them tell us.

4. Make the upstream call us

The best sensor is a POST /api/v1/dags/{dag_id}/dagRuns from whatever produced the data, with the partition in conf. When the producer is another team's system this is a conversation rather than code, but it's usually a short conversation, because they'd also like to stop being asked why their table was late.

What's left

Two ExternalTaskSensors I haven't got to, both in reschedule mode, both with a thirty-minute timeout instead of six hours so that failure is loud and early. Everything else is gone. The DAG files are shorter, the six o'clock worker graph is flat, and when something is late the task that's late is the one that says so.