cosmos-dbt-core

Name: cosmos-dbt-core
Author: astronomer

800installs55forks

How to install cosmos-dbt-core?

astronomer/agents/cosmos-dbt-core

$npx -y skills add astronomer/agents --skill cosmos-dbt-core

Installs into the current project.

Summary

Turns a dbt Core project into an Airflow DAG/TaskGroup using Astronomer Cosmos. Use turning a dbt Core project into an Airflow DAG or TaskGroup with Astronomer Cosmos. Before implementing, verify dbt engine, warehouse, Airflow version, execution environment, DAG vs TaskGroup, and

Files · 1

View on GitHub

SKILL.md

1	# Cosmos + dbt Core: Implementation Checklist
2
3	Execute steps in order. Prefer the simplest configuration that meets the user's constraints.
4
5	> Version note: This skill targets Cosmos 1.11+ and Airflow 3.x. If the user is on Airflow 2.x, adjust imports accordingly (see Appendix A).
6	>
7	> Reference: Latest stable: https://pypi.org/project/astronomer-cosmos/
8
9	> Before starting, confirm: (1) dbt engine = Core (not Fusion → use cosmos-dbt-fusion), (2) warehouse type, (3) Airflow version, (4) execution environment (Airflow env / venv / container), (5) DbtDag vs DbtTaskGroup vs individual operators, (6) manifest availability.
10
11	---
12
13	## 1. Configure Project (ProjectConfig)
14
15	\| Approach \| When to use \| Required param \|
16	\|----------\|-------------\|----------------\|
17	\| Project path \| Files available locally \| `dbt_project_path` \|
18	\| Manifest only \| `dbt_manifest` load \| `manifest_path` + `project_name` \|
19
20	```python
21	from cosmos import ProjectConfig
22
23	_project_config = ProjectConfig(
24	dbt_project_path="/path/to/dbt/project",
25	# manifest_path="/path/to/manifest.json", # for dbt_manifest load mode
26	# project_name="my_project", # if using manifest_path without dbt_project_path
27	# install_dbt_deps=False, # if deps precomputed in CI
28	)
29	```
30
31	## 2. Choose Parsing Strategy (RenderConfig)
32
33	Pick ONE load mode based on constraints:
34
35	\| Load mode \| When to use \| Required inputs \| Constraints \|
36	\|-----------\|-------------\|-----------------\|-------------\|
37	\| `dbt_manifest` \| Large projects; containerized execution; fastest \| `ProjectConfig.manifest_path` \| Remote manifest needs `manifest_conn_id` \|
38	\| `dbt_ls` \| Complex selectors; need dbt-native selection \| dbt installed OR `dbt_executable_path` \| Can also be used with containerized execution \|
39	\| `dbt_ls_file` \| dbt_ls selection without running dbt_ls every parse \| `RenderConfig.dbt_ls_path` \| `select`/`exclude` won't work \|
40	\| `automatic` (default) \| Simple setups; let Cosmos pick \| (none) \| Falls back: manifest → dbt_ls → custom \|
41
42	> CRITICAL: Containerized execution (`DOCKER`/`KUBERNETES`/etc.)
43
44	```python
45	from cosmos import RenderConfig, LoadMode
46
47	_render_config = RenderConfig(
48	load_method=LoadMode.DBT_MANIFEST, # or DBT_LS, DBT_LS_FILE, AUTOMATIC
49	)
50	```
51
52	---
53
54	## 3. Choose Execution Mode (ExecutionConfig)
55
56	> Reference: See [reference/cosmos-config.md](reference/cosmos-config.md#execution-modes-executionconfig) for detailed configuration examples per mode.
57
58	Pick ONE execution mode:
59
60	\| Execution mode \| When to use \| Speed \| Required setup \|
61	\|----------------\|-------------\|-------\|----------------\|
62	\| `WATCHER` \| Fastest; single `dbt build` visibility \| Fastest \| dbt adapter in env OR `dbt_executable_path` or dbt Fusion \|
63	\| `WATCHER_KUBERNETES` \| Fastest isolated method; single `dbt build` visibility \| Fast \| dbt installed in container \|
64	\| `LOCAL` + `DBT_RUNNER` \| dbt + adapter in the same Python installation as Airflow \| Fast \| dbt 1.5+ in `requirements.txt` \|
65	\| `LOCAL` + `SUBPROCESS` \| dbt + adapter available in the Airflow deployment, in an isolated Python installation \| Medium \| `dbt_executable_path` \|
66	\| `AIRFLOW_ASYNC` \| BigQuery + long-running transforms \| Fast \| Airflow ≥2.8; provider deps \|
67	\| `KUBERNETES` \| Isolation between Airflow and dbt \| Medium \| Airflow ≥2.8; provider deps \|
68	\| `VIRTUALENV` \| Can't modify image; runtime venv \| Slower \| `py_requirements` in operator_args \|
69	\| Other containerized approaches \| Support Airflow and dbt isolation \| Medium \| container config \|
70
71	```python
72	from cosmos import ExecutionConfig, ExecutionMode
73
74	_execution_config = ExecutionConfig(
75	execution_mode=ExecutionMode.WATCHER, # or LOCAL, VIRTUALENV, AIRFLOW_ASYNC, KUBERNETES, etc.
76	)
77	```
78
79	---
80
81	## 4. Configure Warehouse Connection (ProfileConfig)
82
83	> Reference: See [reference/cosmos-config.md](reference/cosmos-config.md#profileconfig-warehouse-connection) for detailed ProfileConfig options and all ProfileMapping classes.
84
85	### Option A: Airflow Connection + ProfileMapping (Recommended)
86
87	```python
88	from cosmos import ProfileConfig
89	from cosmos.profiles import SnowflakeUserPasswordProfileMapping
90
91	_profile_config = ProfileConfig(
92	profile_name="default",
93	target_name="dev",
94	profile_mapping=SnowflakeUserPasswordProfileMapping(
95	conn_id="snowflake_default",
96	profile_args={"schema": "my_schema"},
97	),
98	)
99	```
100
101	### Option B: Existing profiles.yml
102
103	> CRITICAL: Do not hardcode secrets; use environment variables.
104
105	```python
106	from cosmos import ProfileConfig
107
108	_profile_config = ProfileConfig(
109	profile_name="my_profile",
110	target_name="dev",
111	profiles_yml_filepath="/path/to/profiles.yml",
112	)
113	`

People who liked this also viewed

6 picks

Type

cosmos-dbt-core

800installs55forks

View on GitHub

How to install cosmos-dbt-core?

astronomer/agents/cosmos-dbt-core

$npx -y skills add astronomer/agents --skill cosmos-dbt-core

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Cosmos + dbt Core: Implementation Checklist
2
3	Execute steps in order. Prefer the simplest configuration that meets the user's constraints.
4
5	> Version note: This skill targets Cosmos 1.11+ and Airflow 3.x. If the user is on Airflow 2.x, adjust imports accordingly (see Appendix A).
6	>
7	> Reference: Latest stable: https://pypi.org/project/astronomer-cosmos/
8
9	> Before starting, confirm: (1) dbt engine = Core (not Fusion → use cosmos-dbt-fusion), (2) warehouse type, (3) Airflow version, (4) execution environment (Airflow env / venv / container), (5) DbtDag vs DbtTaskGroup vs individual operators, (6) manifest availability.
10
11	---
12
13	## 1. Configure Project (ProjectConfig)
14
15	\| Approach \| When to use \| Required param \|
16	\|----------\|-------------\|----------------\|
17	\| Project path \| Files available locally \| `dbt_project_path` \|
18	\| Manifest only \| `dbt_manifest` load \| `manifest_path` + `project_name` \|
19
20	```python
21	from cosmos import ProjectConfig
22
23	_project_config = ProjectConfig(
24	dbt_project_path="/path/to/dbt/project",
25	# manifest_path="/path/to/manifest.json", # for dbt_manifest load mode
26	# project_name="my_project", # if using manifest_path without dbt_project_path
27	# install_dbt_deps=False, # if deps precomputed in CI
28	)
29	```
30
31	## 2. Choose Parsing Strategy (RenderConfig)
32
33	Pick ONE load mode based on constraints:
34
35	\| Load mode \| When to use \| Required inputs \| Constraints \|
36	\|-----------\|-------------\|-----------------\|-------------\|
37	\| `dbt_manifest` \| Large projects; containerized execution; fastest \| `ProjectConfig.manifest_path` \| Remote manifest needs `manifest_conn_id` \|
38	\| `dbt_ls` \| Complex selectors; need dbt-native selection \| dbt installed OR `dbt_executable_path` \| Can also be used with containerized execution \|
39	\| `dbt_ls_file` \| dbt_ls selection without running dbt_ls every parse \| `RenderConfig.dbt_ls_path` \| `select`/`exclude` won't work \|
40	\| `automatic` (default) \| Simple setups; let Cosmos pick \| (none) \| Falls back: manifest → dbt_ls → custom \|
41
42	> CRITICAL: Containerized execution (`DOCKER`/`KUBERNETES`/etc.)
43
44	```python
45	from cosmos import RenderConfig, LoadMode
46
47	_render_config = RenderConfig(
48	load_method=LoadMode.DBT_MANIFEST, # or DBT_LS, DBT_LS_FILE, AUTOMATIC
49	)
50	```
51
52	---
53
54	## 3. Choose Execution Mode (ExecutionConfig)
55
56	> Reference: See [reference/cosmos-config.md](reference/cosmos-config.md#execution-modes-executionconfig) for detailed configuration examples per mode.
57
58	Pick ONE execution mode:
59
60	\| Execution mode \| When to use \| Speed \| Required setup \|
61	\|----------------\|-------------\|-------\|----------------\|
62	\| `WATCHER` \| Fastest; single `dbt build` visibility \| Fastest \| dbt adapter in env OR `dbt_executable_path` or dbt Fusion \|
63	\| `WATCHER_KUBERNETES` \| Fastest isolated method; single `dbt build` visibility \| Fast \| dbt installed in container \|
64	\| `LOCAL` + `DBT_RUNNER` \| dbt + adapter in the same Python installation as Airflow \| Fast \| dbt 1.5+ in `requirements.txt` \|
65	\| `LOCAL` + `SUBPROCESS` \| dbt + adapter available in the Airflow deployment, in an isolated Python installation \| Medium \| `dbt_executable_path` \|
66	\| `AIRFLOW_ASYNC` \| BigQuery + long-running transforms \| Fast \| Airflow ≥2.8; provider deps \|
67	\| `KUBERNETES` \| Isolation between Airflow and dbt \| Medium \| Airflow ≥2.8; provider deps \|
68	\| `VIRTUALENV` \| Can't modify image; runtime venv \| Slower \| `py_requirements` in operator_args \|
69	\| Other containerized approaches \| Support Airflow and dbt isolation \| Medium \| container config \|
70
71	```python
72	from cosmos import ExecutionConfig, ExecutionMode
73
74	_execution_config = ExecutionConfig(
75	execution_mode=ExecutionMode.WATCHER, # or LOCAL, VIRTUALENV, AIRFLOW_ASYNC, KUBERNETES, etc.
76	)
77	```
78
79	---
80
81	## 4. Configure Warehouse Connection (ProfileConfig)
82
83	> Reference: See [reference/cosmos-config.md](reference/cosmos-config.md#profileconfig-warehouse-connection) for detailed ProfileConfig options and all ProfileMapping classes.
84
85	### Option A: Airflow Connection + ProfileMapping (Recommended)
86
87	```python
88	from cosmos import ProfileConfig
89	from cosmos.profiles import SnowflakeUserPasswordProfileMapping
90
91	_profile_config = ProfileConfig(
92	profile_name="default",
93	target_name="dev",
94	profile_mapping=SnowflakeUserPasswordProfileMapping(
95	conn_id="snowflake_default",
96	profile_args={"schema": "my_schema"},
97	),
98	)
99	```
100
101	### Option B: Existing profiles.yml
102
103	> CRITICAL: Do not hardcode secrets; use environment variables.
104
105	```python
106	from cosmos import ProfileConfig
107
108	_profile_config = ProfileConfig(
109	profile_name="my_profile",
110	target_name="dev",
111	profiles_yml_filepath="/path/to/profiles.yml",
112	)
113	`

People who liked this also viewed

6 picks

Type