Skip to content

pleasefixai/data-engineering

v1.0.1Apache-2.0

Guides to explore warehouses, trace data lineage and run Airflow pipelines on Astro.

airflow-state-store

Persists task and asset state across retries and DAG runs using Airflow 3.3's AIP-103 key/value stores (task_state_store, asset_state_store) and the crash-safe ResumableJobMixin. Use when the user asks about task state store, checkpointing in tasks, persisting state across retries, job IDs surviving worker crashes, watermarks, asset metadata, resumable tasks, crash-safe operators, or "what's new in Airflow 3.3". Also use proactively when reading a DAG that uses Variables or XCom for intra-task coordination state (flag the anti-pattern), or when reviewing any DAG that submits a job to an external system (Databricks, Snowflake, BigQuery, Redshift, Spark, dbt Cloud, EMR, AWS Batch, etc.) and waits for it to finish, whether as one submit-and-wait operator or split into submit + sensor/polling tasks; covers wait_for_termination, deferrable, durable, and hand-rolled polling sensors. State-persistence pieces require Airflow 3.3+; submit+poll architecture guidance applies on any version.

Read SKILL.md at the source

Pinned to revision 4e8964e24a5a, so it is the text this page describes rather than whatever the author pushed since.

Files

Every link opens the file at its source, pinned to the revision this page describes.