deploying-airflow

Name: deploying-airflow
Author: astronomer

583installs55forks

How to install deploying-airflow?

astronomer/agents/deploying-airflow

$npx -y skills add astronomer/agents --skill deploying-airflow

Installs into the current project.

Summary

Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.

Files · 1

View on GitHub

SKILL.md

1	# Deploying Airflow
2
3	This skill covers deploying Airflow DAGs and projects to production, whether using Astro (Astronomer's managed platform) or open-source Airflow on Docker Compose or Kubernetes.
4
5	Choosing a path: Astro is a good fit for managed operations and faster CI/CD. For open-source, use Docker Compose for dev and the Helm chart for production.
6
7	---
8
9	## Astro (Astronomer)
10
11	Astro provides CLI commands and GitHub integration for deploying Airflow projects.
12
13	### Deploy Commands
14
15	\| Command \| What It Does \|
16	\|---------\|--------------\|
17	\| `astro deploy` \| Full project deploy — builds Docker image and deploys DAGs \|
18	\| `astro deploy --dags` \| DAG-only deploy — pushes only DAG files (fast, no image build) \|
19	\| `astro deploy --image` \| Image-only deploy — pushes only the Docker image (for multi-repo CI/CD) \|
20	\| `astro deploy --dbt` \| dbt project deploy — deploys a dbt project to run alongside Airflow \|
21
22	### Full Project Deploy
23
24	Builds a Docker image from your Astro project and deploys everything (DAGs, plugins, requirements, packages):
25
26	```bash
27	astro deploy
28	```
29
30	Use this when you've changed `requirements.txt`, `Dockerfile`, `packages.txt`, plugins, or any non-DAG file.
31
32	### DAG-Only Deploy
33
34	Pushes only files in the `dags/` directory without rebuilding the Docker image:
35
36	```bash
37	astro deploy --dags
38	```
39
40	This is significantly faster than a full deploy since it skips the image build. Use this when you've only changed DAG files and haven't modified dependencies or configuration.
41
42	### Image-Only Deploy
43
44	Pushes only the Docker image without updating DAGs:
45
46	```bash
47	astro deploy --image
48	```
49
50	This is useful in multi-repo setups where DAGs are deployed separately from the image, or in CI/CD pipelines that manage image and DAG deploys independently.
51
52	### dbt Project Deploy
53
54	Deploys a dbt project to run with Cosmos on an Astro deployment:
55
56	```bash
57	astro deploy --dbt
58	```
59
60	### GitHub Integration
61
62	Astro supports branch-to-deployment mapping for automated deploys:
63
64	- Map branches to specific deployments (e.g., `main` -> production, `develop` -> staging)
65	- Pushes to mapped branches trigger automatic deploys
66	- Supports DAG-only deploys on merge for faster iteration
67
68	Configure this in the Astro UI under Deployment Settings > CI/CD.
69
70	### CI/CD Patterns
71
72	Common CI/CD strategies on Astro:
73
74	1. DAG-only on feature branches: Use `astro deploy --dags` for fast iteration during development
75	2. Full deploy on main: Use `astro deploy` on merge to main for production releases
76	3. Separate image and DAG pipelines: Use `--image` and `--dags` in separate CI jobs for independent release cycles
77
78	### Deploy Queue
79
80	When multiple deploys are triggered in quick succession, Astro processes them sequentially in a deploy queue. Each deploy completes before the next one starts.
81
82	### Reference
83
84	- [Astro Deploy Documentation](https://www.astronomer.io/docs/astro/deploy-code)
85
86	---
87
88	## Open-Source: Docker Compose
89
90	Deploy Airflow using the official Docker Compose setup. This is recommended for learning and exploration — for production, use Kubernetes with the Helm chart (see below).
91
92	### Prerequisites
93
94	- Docker and Docker Compose v2.14.0+
95	- The official `apache/airflow` Docker image
96
97	### Quick Start
98
99	Download the official Airflow 3 Docker Compose file:
100
101	```bash
102	curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
103	```
104
105	This sets up the full Airflow 3 architecture:
106
107	\| Service \| Purpose \|
108	\|---------\|---------\|
109	\| `airflow-apiserver` \| REST API and UI (port 8080) \|
110	\| `airflow-scheduler` \| Schedules DAG runs \|
111	\| `airflow-dag-processor` \| Parses and processes DAG files \|
112	\| `airflow-worker` \| Executes tasks (CeleryExecutor) \|
113	\| `airflow-triggerer` \| Handles deferrable/async tasks \|
114	\| `postgres` \| Metadata database \|
115	\| `redis` \| Celery message broker \|
116
117	### Minimal Setup
118
119	For a simpler setup with LocalExecutor (no Celery/Redis), create a `docker-compose.yaml`:
120
121	```yaml
122	x-airflow-common: &airflow-common
123	image: apache/airflow:3 # Use the latest Airflow 3.x release
124	environment: &airflow-common-env
125	AIRFLOW__CORE__EXECUTOR: LocalExecutor
126	AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
127	AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
128	AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags
129	volumes:
130	- ./dags:/opt/airflow/dags
131	- ./logs:/opt/airflow/logs
132	- ./plugins:/opt/airflow/plugins
133	depends_on:
134	postgres:
135	condition: service_healthy
136
137	services:
138	postgres:
139	image: postgres:16
140	environment:
141	POSTGRES_USER: airflow
142	POSTGRES_PASSWORD: airflow
143	POSTGRES_DB: airflow
144	volumes:
145	- postgres-db-volume:/var/lib/postgresql/data
146	healthcheck:
147	test: ["CMD", "pg_isready"

People who liked this also viewed

6 picks

Type

deploying-airflow

583installs55forks

View on GitHub

How to install deploying-airflow?

astronomer/agents/deploying-airflow

$npx -y skills add astronomer/agents --skill deploying-airflow

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Deploying Airflow
2
3	This skill covers deploying Airflow DAGs and projects to production, whether using Astro (Astronomer's managed platform) or open-source Airflow on Docker Compose or Kubernetes.
4
5	Choosing a path: Astro is a good fit for managed operations and faster CI/CD. For open-source, use Docker Compose for dev and the Helm chart for production.
6
7	---
8
9	## Astro (Astronomer)
10
11	Astro provides CLI commands and GitHub integration for deploying Airflow projects.
12
13	### Deploy Commands
14
15	\| Command \| What It Does \|
16	\|---------\|--------------\|
17	\| `astro deploy` \| Full project deploy — builds Docker image and deploys DAGs \|
18	\| `astro deploy --dags` \| DAG-only deploy — pushes only DAG files (fast, no image build) \|
19	\| `astro deploy --image` \| Image-only deploy — pushes only the Docker image (for multi-repo CI/CD) \|
20	\| `astro deploy --dbt` \| dbt project deploy — deploys a dbt project to run alongside Airflow \|
21
22	### Full Project Deploy
23
24	Builds a Docker image from your Astro project and deploys everything (DAGs, plugins, requirements, packages):
25
26	```bash
27	astro deploy
28	```
29
30	Use this when you've changed `requirements.txt`, `Dockerfile`, `packages.txt`, plugins, or any non-DAG file.
31
32	### DAG-Only Deploy
33
34	Pushes only files in the `dags/` directory without rebuilding the Docker image:
35
36	```bash
37	astro deploy --dags
38	```
39
40	This is significantly faster than a full deploy since it skips the image build. Use this when you've only changed DAG files and haven't modified dependencies or configuration.
41
42	### Image-Only Deploy
43
44	Pushes only the Docker image without updating DAGs:
45
46	```bash
47	astro deploy --image
48	```
49
50	This is useful in multi-repo setups where DAGs are deployed separately from the image, or in CI/CD pipelines that manage image and DAG deploys independently.
51
52	### dbt Project Deploy
53
54	Deploys a dbt project to run with Cosmos on an Astro deployment:
55
56	```bash
57	astro deploy --dbt
58	```
59
60	### GitHub Integration
61
62	Astro supports branch-to-deployment mapping for automated deploys:
63
64	- Map branches to specific deployments (e.g., `main` -> production, `develop` -> staging)
65	- Pushes to mapped branches trigger automatic deploys
66	- Supports DAG-only deploys on merge for faster iteration
67
68	Configure this in the Astro UI under Deployment Settings > CI/CD.
69
70	### CI/CD Patterns
71
72	Common CI/CD strategies on Astro:
73
74	1. DAG-only on feature branches: Use `astro deploy --dags` for fast iteration during development
75	2. Full deploy on main: Use `astro deploy` on merge to main for production releases
76	3. Separate image and DAG pipelines: Use `--image` and `--dags` in separate CI jobs for independent release cycles
77
78	### Deploy Queue
79
80	When multiple deploys are triggered in quick succession, Astro processes them sequentially in a deploy queue. Each deploy completes before the next one starts.
81
82	### Reference
83
84	- [Astro Deploy Documentation](https://www.astronomer.io/docs/astro/deploy-code)
85
86	---
87
88	## Open-Source: Docker Compose
89
90	Deploy Airflow using the official Docker Compose setup. This is recommended for learning and exploration — for production, use Kubernetes with the Helm chart (see below).
91
92	### Prerequisites
93
94	- Docker and Docker Compose v2.14.0+
95	- The official `apache/airflow` Docker image
96
97	### Quick Start
98
99	Download the official Airflow 3 Docker Compose file:
100
101	```bash
102	curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
103	```
104
105	This sets up the full Airflow 3 architecture:
106
107	\| Service \| Purpose \|
108	\|---------\|---------\|
109	\| `airflow-apiserver` \| REST API and UI (port 8080) \|
110	\| `airflow-scheduler` \| Schedules DAG runs \|
111	\| `airflow-dag-processor` \| Parses and processes DAG files \|
112	\| `airflow-worker` \| Executes tasks (CeleryExecutor) \|
113	\| `airflow-triggerer` \| Handles deferrable/async tasks \|
114	\| `postgres` \| Metadata database \|
115	\| `redis` \| Celery message broker \|
116
117	### Minimal Setup
118
119	For a simpler setup with LocalExecutor (no Celery/Redis), create a `docker-compose.yaml`:
120
121	```yaml
122	x-airflow-common: &airflow-common
123	image: apache/airflow:3 # Use the latest Airflow 3.x release
124	environment: &airflow-common-env
125	AIRFLOW__CORE__EXECUTOR: LocalExecutor
126	AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
127	AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
128	AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags
129	volumes:
130	- ./dags:/opt/airflow/dags
131	- ./logs:/opt/airflow/logs
132	- ./plugins:/opt/airflow/plugins
133	depends_on:
134	postgres:
135	condition: service_healthy
136
137	services:
138	postgres:
139	image: postgres:16
140	environment:
141	POSTGRES_USER: airflow
142	POSTGRES_PASSWORD: airflow
143	POSTGRES_DB: airflow
144	volumes:
145	- postgres-db-volume:/var/lib/postgresql/data
146	healthcheck:
147	test: ["CMD", "pg_isready"

People who liked this also viewed

6 picks

Type