airflow-and-workflow-orchestration

Name: airflow-and-workflow-orchestration
Author: vaquarkhan

19stars4forks

How to install airflow-and-workflow-orchestration?

vaquarkhan/data-engineering-agent-skills/airflow-and-workflow-orchestration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill airflow-and-workflow-orchestration

Installs into the current project.

Summary

Guides agents through workflow orchestration design and operation across Airflow-style DAGs, cloud-native schedulers, and event-driven pipeline control planes. Use when building or modifying workflow dependencies, retries, triggers, sensors, SLAs, or cross-system pipeline coordin

Files · 1

View on GitHub

SKILL.md

1	# Airflow And Workflow Orchestration
2
3	## Overview
4
5	Use this skill when the problem is orchestration rather than transformation logic. It helps agents design `Airflow`, `MWAA`, `Cloud Composer`, `Azure Data Factory`, `Step Functions`, `Google Cloud Workflows`, or `Databricks Workflows` patterns with explicit dependencies, retries, ownership, backfills, and publish-safe cutover behavior.
6
7	## When to Use
8
9	- building or modifying `Airflow` DAGs
10	- choosing between scheduler-driven, event-driven, or platform-native orchestration
11	- designing scheduling, sensors, task dependencies, or retry policy
12	- coordinating ingestion, transformation, quality checks, and publish steps
13	- changing backfill, catchup, or SLA behavior
14
15	Do not use this as a substitute for the underlying processing skill. Orchestration coordinates work; it does not define the compute logic itself.
16
17	## Workflow
18
19	1. Define the workflow contract.
20	Capture:
21	- owner
22	- schedule or trigger mode
23	- upstream and downstream dependencies
24	- task boundaries
25	- success and failure signals
26
27	2. Choose the orchestration model before choosing the product.
28	Decide whether the workload is:
29	- scheduler-driven
30	- event-driven
31	- metadata-driven fan-out
32	- platform-native inside a lakehouse
33
34	3. Separate orchestration concerns from processing concerns.
35	Workflow tasks should call well-defined jobs, not hide business logic in orchestration code.
36
37	4. Design retries and timeouts deliberately.
38	Account for:
39	- idempotency
40	- duplicate writes
41	- sensor cost
42	- late-arriving upstream data
43	- alert routing
44
45	5. Make backfill behavior explicit.
46	Decide how catchup, reruns, and historical windows behave before enabling them.
47
48	6. Gate publish steps on validation.
49	A successful task chain is not enough if downstream tables fail quality checks.
50
51	## Service Selection Hints
52
53	- `AWS`: prefer `MWAA` for dependency-rich DAGs, `Step Functions` for branching and service coordination, and lightweight event triggers for arrival-based flows.
54	- `Azure`: prefer `Azure Data Factory` or `Synapse Pipelines` for connector-heavy and parameterized workflows, and `Databricks Workflows` when execution stays in `Databricks`.
55	- `GCP`: prefer `Cloud Composer` for DAGs, `Google Cloud Workflows` for API and service coordination, and `Cloud Scheduler` plus `Pub/Sub` for simple triggers.
56	- `Databricks`: prefer `Databricks Workflows` or `Delta Live Tables` for platform-native jobs, and use an external orchestrator when dependencies cross platform boundaries.
57	- Load `references/pipeline-orchestration-patterns.md` when selecting or reviewing the orchestration model, not only the syntax of one tool.
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "We can keep the transformation logic inside the DAG file." \| That creates brittle orchestration code and harder testing and reuse. \|
64	\| "Retries are always safe." \| Retries on non-idempotent tasks can duplicate data or corrupt publish layers. \|
65	\| "Catchup will handle backfills automatically." \| Historical reprocessing usually needs different safeguards than normal schedule runs. \|
66	\| "The platform-default orchestrator is automatically the right one." \| Tool choice should follow workload shape, cross-system boundaries, and recovery needs. \|
67
68	## Red Flags
69
70	- business logic is embedded in scheduler wiring
71	- orchestrator choice is driven by habit rather than workflow shape
72	- task ownership and alerts are undefined
73	- sensors poll without clear timeout or cost awareness
74	- publish tasks run without validation gates
75	- backfills rely on default catchup behavior with no review
76
77	## Verification
78
79	- [ ] DAG responsibilities are separated from compute logic
80	- [ ] Orchestrator choice matches the workflow shape and platform boundary
81	- [ ] Scheduling, retries, alerts, and dependencies are explicit
82	- [ ] Backfill and catchup behavior are documented
83	- [ ] Publish sequencing depends on validation, not just task completion

People who liked this also viewed

6 picks

Type

airflow-and-workflow-orchestration

19stars4forks

View on GitHub

How to install airflow-and-workflow-orchestration?

vaquarkhan/data-engineering-agent-skills/airflow-and-workflow-orchestration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill airflow-and-workflow-orchestration

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Airflow And Workflow Orchestration
2
3	## Overview
4
5	Use this skill when the problem is orchestration rather than transformation logic. It helps agents design `Airflow`, `MWAA`, `Cloud Composer`, `Azure Data Factory`, `Step Functions`, `Google Cloud Workflows`, or `Databricks Workflows` patterns with explicit dependencies, retries, ownership, backfills, and publish-safe cutover behavior.
6
7	## When to Use
8
9	- building or modifying `Airflow` DAGs
10	- choosing between scheduler-driven, event-driven, or platform-native orchestration
11	- designing scheduling, sensors, task dependencies, or retry policy
12	- coordinating ingestion, transformation, quality checks, and publish steps
13	- changing backfill, catchup, or SLA behavior
14
15	Do not use this as a substitute for the underlying processing skill. Orchestration coordinates work; it does not define the compute logic itself.
16
17	## Workflow
18
19	1. Define the workflow contract.
20	Capture:
21	- owner
22	- schedule or trigger mode
23	- upstream and downstream dependencies
24	- task boundaries
25	- success and failure signals
26
27	2. Choose the orchestration model before choosing the product.
28	Decide whether the workload is:
29	- scheduler-driven
30	- event-driven
31	- metadata-driven fan-out
32	- platform-native inside a lakehouse
33
34	3. Separate orchestration concerns from processing concerns.
35	Workflow tasks should call well-defined jobs, not hide business logic in orchestration code.
36
37	4. Design retries and timeouts deliberately.
38	Account for:
39	- idempotency
40	- duplicate writes
41	- sensor cost
42	- late-arriving upstream data
43	- alert routing
44
45	5. Make backfill behavior explicit.
46	Decide how catchup, reruns, and historical windows behave before enabling them.
47
48	6. Gate publish steps on validation.
49	A successful task chain is not enough if downstream tables fail quality checks.
50
51	## Service Selection Hints
52
53	- `AWS`: prefer `MWAA` for dependency-rich DAGs, `Step Functions` for branching and service coordination, and lightweight event triggers for arrival-based flows.
54	- `Azure`: prefer `Azure Data Factory` or `Synapse Pipelines` for connector-heavy and parameterized workflows, and `Databricks Workflows` when execution stays in `Databricks`.
55	- `GCP`: prefer `Cloud Composer` for DAGs, `Google Cloud Workflows` for API and service coordination, and `Cloud Scheduler` plus `Pub/Sub` for simple triggers.
56	- `Databricks`: prefer `Databricks Workflows` or `Delta Live Tables` for platform-native jobs, and use an external orchestrator when dependencies cross platform boundaries.
57	- Load `references/pipeline-orchestration-patterns.md` when selecting or reviewing the orchestration model, not only the syntax of one tool.
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "We can keep the transformation logic inside the DAG file." \| That creates brittle orchestration code and harder testing and reuse. \|
64	\| "Retries are always safe." \| Retries on non-idempotent tasks can duplicate data or corrupt publish layers. \|
65	\| "Catchup will handle backfills automatically." \| Historical reprocessing usually needs different safeguards than normal schedule runs. \|
66	\| "The platform-default orchestrator is automatically the right one." \| Tool choice should follow workload shape, cross-system boundaries, and recovery needs. \|
67
68	## Red Flags
69
70	- business logic is embedded in scheduler wiring
71	- orchestrator choice is driven by habit rather than workflow shape
72	- task ownership and alerts are undefined
73	- sensors poll without clear timeout or cost awareness
74	- publish tasks run without validation gates
75	- backfills rely on default catchup behavior with no review
76
77	## Verification
78
79	- [ ] DAG responsibilities are separated from compute logic
80	- [ ] Orchestrator choice matches the workflow shape and platform boundary
81	- [ ] Scheduling, retries, alerts, and dependencies are explicit
82	- [ ] Backfill and catchup behavior are documented
83	- [ ] Publish sequencing depends on validation, not just task completion

People who liked this also viewed

6 picks

Type