dag-factory

Name: dag-factory
Author: astronomer

230installs55forks

How to install dag-factory?

astronomer/agents/dag-factory

$npx -y skills add astronomer/agents --skill dag-factory

Installs into the current project.

Summary

Authors Apache Airflow DAGs declaratively from dag-factory YAML configs. Use when building DAGs declaratively from YAML via dag-factory; creating/editing dag-factory templates/YAML configs,reating/editing dag-factory YAML configs, defaults, dynamic tasks, datasets, or callbacks;

Files · 1

View on GitHub

SKILL.md

1	# DAG Factory
2
3	You are helping a user build Apache Airflow DAGs declaratively with dag-factory, a library that turns YAML configuration files into Airflow DAGs. Execute steps in order and prefer the simplest configuration that meets the user's needs.
4
5	> Package: `dag-factory` on PyPI
6	> Repo: https://github.com/astronomer/dag-factory
7	> Docs: https://astronomer.github.io/dag-factory/latest/
8	> Targets: dag-factory v1.0+ only. For pre-1.0 projects, see [reference/migration.md](reference/migration.md) before applying any guidance from this skill.
9	> Requires: Python 3.10+, Airflow 2.4+ (Airflow 3 supported)
10
11	## Before Starting
12
13	Confirm with the user:
14	1. Airflow version ≥2.4
15	2. Python version ≥3.10
16	3. dag-factory version: this skill targets v1.0+. If the project is on <1.0, follow [reference/migration.md](reference/migration.md) to upgrade before continuing.
17	4. Use case: dag-factory is for declarative, low-code DAG authoring. If the user needs reusable, validated Pythonic templates with Pydantic, suggest blueprint instead. If they need full Python flexibility, suggest the authoring-dags skill.
18
19	---
20
21	## Determine What the User Needs
22
23	\| User Request \| Action \|
24	\|--------------\|--------\|
25	\| "Create a YAML DAG" / "Convert this Python DAG to YAML" \| Go to Defining a DAG in YAML \|
26	\| "Set up dag-factory in my project" \| Go to Project Setup \|
27	\| "Share defaults across DAGs" / "Set start_date once" \| Go to Defaults \|
28	\| "Use a custom operator" / "Use KPO / Slack / Snowflake" \| Go to Custom & Provider Operators \|
29	\| "Dynamic / mapped tasks" / "expand / partial" \| Go to Dynamic Task Mapping \|
30	\| "Schedule on dataset" / "Outlets and inlets" \| Go to Datasets \|
31	\| "Add a callback" / "Slack on failure" \| Go to Callbacks \|
32	\| "Use a timetable" / "datetime in YAML" / "timedelta in YAML" \| Go to Custom Python Objects (`__type__`) \|
33	\| "Lint my YAML" / "Validate" \| Go to Validation Commands \|
34	\| "Convert Airflow 2 YAML to Airflow 3" \| Go to Validation Commands (`dagfactory convert`) \|
35	\| "Migrate from dag-factory <1.0" \| See [reference/migration.md](reference/migration.md) \|
36	\| dag-factory errors / troubleshooting \| Go to Troubleshooting \|
37
38	---
39
40	## Project Setup
41
42	### 1. Install the Package
43
44	Add to `requirements.txt`:
45
46	```
47	dag-factory>=1.0.0
48	```
49
50	dag-factory does not install Airflow providers automatically. Install any provider packages your YAML references (e.g., `apache-airflow-providers-slack`, `apache-airflow-providers-cncf-kubernetes`).
51
52	### 2. Create the Loader
53
54	Create `dags/load_dags.py` so Airflow's DAG processor will pick it up:
55
56	```python
57	import os
58	from pathlib import Path
59
60	from dagfactory import load_yaml_dags
61
62	CONFIG_ROOT_DIR = Path(os.getenv("CONFIG_ROOT_DIR", "/usr/local/airflow/dags/"))
63
64	# Option A: load every .yml / .yaml under a folder
65	load_yaml_dags(globals_dict=globals(), dags_folder=str(CONFIG_ROOT_DIR))
66
67	# Option B: load a single file
68	# load_yaml_dags(globals_dict=globals(), config_filepath=str(CONFIG_ROOT_DIR / "my_dag.yml"))
69
70	# Option C: load from an in-Python dict
71	# load_yaml_dags(globals_dict=globals(), config_dict={...})
72	```
73
74	`globals_dict=globals()` is required so generated DAG objects are registered into the module namespace where Airflow can discover them.
75
76	### 3. Verify Installation
77
78	```bash
79	dagfactory --version
80	```
81
82	---
83
84	## Defining a DAG in YAML
85
86	Each top-level YAML key (other than `default`) defines a DAG. The key becomes the `dag_id`. Use the list format for `tasks` and `task_groups` — it is the recommended format since v1.0.0.
87
88	```yaml
89	# dags/example_dag_factory.yml
90	default:
91	default_args:
92	start_date: 2024-11-11
93
94	basic_example_dag:
95	default_args:
96	owner: "custom_owner"
97	description: "this is an example dag"
98	schedule: "0 3 * * *"
99	catchup: false
100	task_groups:
101	- group_name: "example_task_group"
102	tooltip: "this is an example task group"
103	dependencies: [task_1]
104	tasks:
105	- task_id: "task_1"
106	operator: airflow.operators.bash.BashOperator
107	bash_command: "echo 1"
108	- task_id: "task_2"
109	operator: airflow.operators.bash.BashOperator
110	bash_command: "echo 2"
111	dependencies: [task_1]
112	- task_id: "task_3"
113	operator: airflow.operators.bash.BashOperator
114	bash_command: "echo 3"
115	dependencies: [task_1]
116	task_group_name: "example_task_group"
117	```
118
119	### Key Fields
120
121	\| Field \| Where \| Purpose \|
122	\|-------\|-------\|---------\|
123	\| `default` \| top-level \| Shared DAG-level args applied to every DAG in this file \|
124	\| `default_args` \| DAG or `default` block \| Standard

People who liked this also viewed

6 picks

Type

dag-factory

230installs55forks

View on GitHub

How to install dag-factory?

astronomer/agents/dag-factory

$npx -y skills add astronomer/agents --skill dag-factory

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# DAG Factory
2
3	You are helping a user build Apache Airflow DAGs declaratively with dag-factory, a library that turns YAML configuration files into Airflow DAGs. Execute steps in order and prefer the simplest configuration that meets the user's needs.
4
5	> Package: `dag-factory` on PyPI
6	> Repo: https://github.com/astronomer/dag-factory
7	> Docs: https://astronomer.github.io/dag-factory/latest/
8	> Targets: dag-factory v1.0+ only. For pre-1.0 projects, see [reference/migration.md](reference/migration.md) before applying any guidance from this skill.
9	> Requires: Python 3.10+, Airflow 2.4+ (Airflow 3 supported)
10
11	## Before Starting
12
13	Confirm with the user:
14	1. Airflow version ≥2.4
15	2. Python version ≥3.10
16	3. dag-factory version: this skill targets v1.0+. If the project is on <1.0, follow [reference/migration.md](reference/migration.md) to upgrade before continuing.
17	4. Use case: dag-factory is for declarative, low-code DAG authoring. If the user needs reusable, validated Pythonic templates with Pydantic, suggest blueprint instead. If they need full Python flexibility, suggest the authoring-dags skill.
18
19	---
20
21	## Determine What the User Needs
22
23	\| User Request \| Action \|
24	\|--------------\|--------\|
25	\| "Create a YAML DAG" / "Convert this Python DAG to YAML" \| Go to Defining a DAG in YAML \|
26	\| "Set up dag-factory in my project" \| Go to Project Setup \|
27	\| "Share defaults across DAGs" / "Set start_date once" \| Go to Defaults \|
28	\| "Use a custom operator" / "Use KPO / Slack / Snowflake" \| Go to Custom & Provider Operators \|
29	\| "Dynamic / mapped tasks" / "expand / partial" \| Go to Dynamic Task Mapping \|
30	\| "Schedule on dataset" / "Outlets and inlets" \| Go to Datasets \|
31	\| "Add a callback" / "Slack on failure" \| Go to Callbacks \|
32	\| "Use a timetable" / "datetime in YAML" / "timedelta in YAML" \| Go to Custom Python Objects (`__type__`) \|
33	\| "Lint my YAML" / "Validate" \| Go to Validation Commands \|
34	\| "Convert Airflow 2 YAML to Airflow 3" \| Go to Validation Commands (`dagfactory convert`) \|
35	\| "Migrate from dag-factory <1.0" \| See [reference/migration.md](reference/migration.md) \|
36	\| dag-factory errors / troubleshooting \| Go to Troubleshooting \|
37
38	---
39
40	## Project Setup
41
42	### 1. Install the Package
43
44	Add to `requirements.txt`:
45
46	```
47	dag-factory>=1.0.0
48	```
49
50	dag-factory does not install Airflow providers automatically. Install any provider packages your YAML references (e.g., `apache-airflow-providers-slack`, `apache-airflow-providers-cncf-kubernetes`).
51
52	### 2. Create the Loader
53
54	Create `dags/load_dags.py` so Airflow's DAG processor will pick it up:
55
56	```python
57	import os
58	from pathlib import Path
59
60	from dagfactory import load_yaml_dags
61
62	CONFIG_ROOT_DIR = Path(os.getenv("CONFIG_ROOT_DIR", "/usr/local/airflow/dags/"))
63
64	# Option A: load every .yml / .yaml under a folder
65	load_yaml_dags(globals_dict=globals(), dags_folder=str(CONFIG_ROOT_DIR))
66
67	# Option B: load a single file
68	# load_yaml_dags(globals_dict=globals(), config_filepath=str(CONFIG_ROOT_DIR / "my_dag.yml"))
69
70	# Option C: load from an in-Python dict
71	# load_yaml_dags(globals_dict=globals(), config_dict={...})
72	```
73
74	`globals_dict=globals()` is required so generated DAG objects are registered into the module namespace where Airflow can discover them.
75
76	### 3. Verify Installation
77
78	```bash
79	dagfactory --version
80	```
81
82	---
83
84	## Defining a DAG in YAML
85
86	Each top-level YAML key (other than `default`) defines a DAG. The key becomes the `dag_id`. Use the list format for `tasks` and `task_groups` — it is the recommended format since v1.0.0.
87
88	```yaml
89	# dags/example_dag_factory.yml
90	default:
91	default_args:
92	start_date: 2024-11-11
93
94	basic_example_dag:
95	default_args:
96	owner: "custom_owner"
97	description: "this is an example dag"
98	schedule: "0 3 * * *"
99	catchup: false
100	task_groups:
101	- group_name: "example_task_group"
102	tooltip: "this is an example task group"
103	dependencies: [task_1]
104	tasks:
105	- task_id: "task_1"
106	operator: airflow.operators.bash.BashOperator
107	bash_command: "echo 1"
108	- task_id: "task_2"
109	operator: airflow.operators.bash.BashOperator
110	bash_command: "echo 2"
111	dependencies: [task_1]
112	- task_id: "task_3"
113	operator: airflow.operators.bash.BashOperator
114	bash_command: "echo 3"
115	dependencies: [task_1]
116	task_group_name: "example_task_group"
117	```
118
119	### Key Fields
120
121	\| Field \| Where \| Purpose \|
122	\|-------\|-------\|---------\|
123	\| `default` \| top-level \| Shared DAG-level args applied to every DAG in this file \|
124	\| `default_args` \| DAG or `default` block \| Standard

People who liked this also viewed

6 picks

Type