migrating-airflow-2-to-3

Name: migrating-airflow-2-to-3
Author: astronomer

932installs55forks

How to install migrating-airflow-2-to-3?

astronomer/agents/migrating-airflow-2-to-3

$npx -y skills add astronomer/agents --skill migrating-airflow-2-to-3

Installs into the current project.

Summary

Guide for migrating Apache Airflow 2.x projects to Airflow 3.x. Use when the user mentions Airflow 3 migration, upgrade, compatibility issues, breaking changes, or wants to modernize their Airflow codebase. If you detect Airflow 2.x code that needs migration, prompt the user and

Files · 1

View on GitHub

SKILL.md

1	# Airflow 2 to 3 Migration
2
3	This skill helps migrate Airflow 2.x DAG code to Airflow 3.x, focusing on code changes (imports, operators, hooks, context, API usage).
4
5	Important: Before migrating to Airflow 3, strongly recommend upgrading to Airflow 2.11 first, then to at least Airflow 3.0.11 (ideally directly to 3.1). Other upgrade paths would make rollbacks impossible. See: https://www.astronomer.io/docs/astro/airflow3/upgrade-af3#upgrade-your-airflow-2-deployment-to-airflow-3. Additionally, early 3.0 versions have many bugs - 3.1 provides a much better experience.
6
7	## Migration at a Glance
8
9	1. Run Ruff's Airflow migration rules to auto-fix detectable issues (AIR30/AIR301/AIR302/AIR31/AIR311/AIR312).
10	- `ruff check --preview --select AIR --fix --unsafe-fixes .`
11	2. Scan for remaining issues using the manual search checklist in [reference/migration-checklist.md](reference/migration-checklist.md).
12	- Focus on: direct metadata DB access, legacy imports, scheduling/context keys, XCom pickling, datasets-to-assets, REST API/auth, plugins, and file paths.
13	- Hard behavior/config gotchas to explicitly review:
14	- Cron scheduling semantics: consider `AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVAL=True` if you need Airflow 2-style cron data intervals.
15	- `.airflowignore` syntax changed from regexp to glob; set `AIRFLOW__CORE__DAG_IGNORE_FILE_SYNTAX=regexp` if you must keep regexp behavior.
16	- OAuth callback URLs add an `/auth/` prefix (e.g. `/auth/oauth-authorized/google`).
17	- Shared utility imports: Bare imports like `import common` from `dags/common/` no longer work on Astro. Use fully qualified imports: `import dags.common`.
18	3. Plan changes per file and issue type:
19	- Fix imports - update operators/hooks/providers - refactor metadata access to using the Airflow client instead of direct access - fix use of outdated context variables - fix scheduling logic.
20	4. Implement changes incrementally, re-running Ruff and code searches after each major change.
21	5. Explain changes to the user and caution them to test any updated logic such as refactored metadata, scheduling logic and use of the Airflow context.
22
23	---
24
25	## Architecture & Metadata DB Access
26
27	Airflow 3 changes how components talk to the metadata database:
28
29	- Workers no longer connect directly to the metadata DB.
30	- Task code runs via the Task Execution API exposed by the API server.
31	- The DAG processor runs as an independent process separate from the scheduler.
32	- The Triggerer uses the task execution mechanism via an in-process API server.
33
34	Trigger implementation gotcha: If a trigger calls hooks synchronously inside the asyncio event loop, it may fail or block. Prefer calling hooks via `sync_to_async(...)` (or otherwise ensure hook calls are async-safe).
35
36	Key code impact: Task code can still import ORM sessions/models, but any attempt to use them to talk to the metadata DB will fail with:
37
38	```text
39	RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.x
40	```
41
42	### Patterns to search for
43
44	When scanning DAGs, custom operators, and `@task` functions, look for:
45
46	- Session helpers: `provide_session`, `create_session`, `@provide_session`
47	- Sessions from settings: `from airflow.settings import Session`
48	- Engine access: `from airflow.settings import engine`
49	- ORM usage with models: `session.query(DagModel)...`, `session.query(DagRun)...`
50
51	### Replacement: Airflow Python client
52
53	Preferred for rich metadata access patterns. Add to `requirements.txt`:
54
55	```text
56	apache-airflow-client==<your-airflow-runtime-version>
57	```
58
59	Example usage:
60
61	```python
62	import os
63	from airflow.sdk import BaseOperator
64	import airflow_client.client
65	from airflow_client.client.api.dag_api import DAGApi
66
67	_HOST = os.getenv("AIRFLOW__API__BASE_URL", "https://<your-org>.astronomer.run/<deployment>/")
68	_TOKEN = os.getenv("DEPLOYMENT_API_TOKEN")
69
70	class ListDagsOperator(BaseOperator):
71	def execute(self, context):
72	config = airflow_client.client.Configuration(host=_HOST, access_token=_TOKEN)
73	with airflow_client.client.ApiClient(config) as api_client:
74	dag_api = DAGApi(api_client)
75	dags = dag_api.get_dags(limit=10)
76	self.log.info("Found %d DAGs", len(dags.dags))
77	```
78
79	### Replacement: Direct REST API calls
80
81	For simple cases, call the REST API directly using `requests`:
82
83	```python
84	fr

People who liked this also viewed

6 picks

Type

migrating-airflow-2-to-3

932installs55forks

View on GitHub

How to install migrating-airflow-2-to-3?

astronomer/agents/migrating-airflow-2-to-3

$npx -y skills add astronomer/agents --skill migrating-airflow-2-to-3

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Airflow 2 to 3 Migration
2
3	This skill helps migrate Airflow 2.x DAG code to Airflow 3.x, focusing on code changes (imports, operators, hooks, context, API usage).
4
5	Important: Before migrating to Airflow 3, strongly recommend upgrading to Airflow 2.11 first, then to at least Airflow 3.0.11 (ideally directly to 3.1). Other upgrade paths would make rollbacks impossible. See: https://www.astronomer.io/docs/astro/airflow3/upgrade-af3#upgrade-your-airflow-2-deployment-to-airflow-3. Additionally, early 3.0 versions have many bugs - 3.1 provides a much better experience.
6
7	## Migration at a Glance
8
9	1. Run Ruff's Airflow migration rules to auto-fix detectable issues (AIR30/AIR301/AIR302/AIR31/AIR311/AIR312).
10	- `ruff check --preview --select AIR --fix --unsafe-fixes .`
11	2. Scan for remaining issues using the manual search checklist in [reference/migration-checklist.md](reference/migration-checklist.md).
12	- Focus on: direct metadata DB access, legacy imports, scheduling/context keys, XCom pickling, datasets-to-assets, REST API/auth, plugins, and file paths.
13	- Hard behavior/config gotchas to explicitly review:
14	- Cron scheduling semantics: consider `AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVAL=True` if you need Airflow 2-style cron data intervals.
15	- `.airflowignore` syntax changed from regexp to glob; set `AIRFLOW__CORE__DAG_IGNORE_FILE_SYNTAX=regexp` if you must keep regexp behavior.
16	- OAuth callback URLs add an `/auth/` prefix (e.g. `/auth/oauth-authorized/google`).
17	- Shared utility imports: Bare imports like `import common` from `dags/common/` no longer work on Astro. Use fully qualified imports: `import dags.common`.
18	3. Plan changes per file and issue type:
19	- Fix imports - update operators/hooks/providers - refactor metadata access to using the Airflow client instead of direct access - fix use of outdated context variables - fix scheduling logic.
20	4. Implement changes incrementally, re-running Ruff and code searches after each major change.
21	5. Explain changes to the user and caution them to test any updated logic such as refactored metadata, scheduling logic and use of the Airflow context.
22
23	---
24
25	## Architecture & Metadata DB Access
26
27	Airflow 3 changes how components talk to the metadata database:
28
29	- Workers no longer connect directly to the metadata DB.
30	- Task code runs via the Task Execution API exposed by the API server.
31	- The DAG processor runs as an independent process separate from the scheduler.
32	- The Triggerer uses the task execution mechanism via an in-process API server.
33
34	Trigger implementation gotcha: If a trigger calls hooks synchronously inside the asyncio event loop, it may fail or block. Prefer calling hooks via `sync_to_async(...)` (or otherwise ensure hook calls are async-safe).
35
36	Key code impact: Task code can still import ORM sessions/models, but any attempt to use them to talk to the metadata DB will fail with:
37
38	```text
39	RuntimeError: Direct database access via the ORM is not allowed in Airflow 3.x
40	```
41
42	### Patterns to search for
43
44	When scanning DAGs, custom operators, and `@task` functions, look for:
45
46	- Session helpers: `provide_session`, `create_session`, `@provide_session`
47	- Sessions from settings: `from airflow.settings import Session`
48	- Engine access: `from airflow.settings import engine`
49	- ORM usage with models: `session.query(DagModel)...`, `session.query(DagRun)...`
50
51	### Replacement: Airflow Python client
52
53	Preferred for rich metadata access patterns. Add to `requirements.txt`:
54
55	```text
56	apache-airflow-client==<your-airflow-runtime-version>
57	```
58
59	Example usage:
60
61	```python
62	import os
63	from airflow.sdk import BaseOperator
64	import airflow_client.client
65	from airflow_client.client.api.dag_api import DAGApi
66
67	_HOST = os.getenv("AIRFLOW__API__BASE_URL", "https://<your-org>.astronomer.run/<deployment>/")
68	_TOKEN = os.getenv("DEPLOYMENT_API_TOKEN")
69
70	class ListDagsOperator(BaseOperator):
71	def execute(self, context):
72	config = airflow_client.client.Configuration(host=_HOST, access_token=_TOKEN)
73	with airflow_client.client.ApiClient(config) as api_client:
74	dag_api = DAGApi(api_client)
75	dags = dag_api.get_dags(limit=10)
76	self.log.info("Found %d DAGs", len(dags.dags))
77	```
78
79	### Replacement: Direct REST API calls
80
81	For simple cases, call the REST API directly using `requests`:
82
83	```python
84	fr

People who liked this also viewed

6 picks

Type