warehouse-init

Name: warehouse-init
Author: astronomer

662installs55forks

How to install warehouse-init?

astronomer/agents/warehouse-init

$npx -y skills add astronomer/agents --skill warehouse-init

Installs into the current project.

Summary

Initialize warehouse schema discovery. Generates .astro/warehouse.md with all table metadata for instant lookups. Run once per project, refresh when schema changes. Use when user says "/astronomer-data:warehouse-init" or asks to set up data discovery.

Files · 1

View on GitHub

SKILL.md

1	# Initialize Warehouse Schema
2
3	Generate a comprehensive, user-editable schema reference file for the data warehouse.
4
5	Scripts: `../analyzing-data/scripts/` — All CLI commands below are relative to the `analyzing-data` skill's directory. Before running any `scripts/cli.py` command, `cd` to `../analyzing-data/` relative to this file.
6
7	## What This Does
8
9	1. Discovers all databases, schemas, tables, and columns from the warehouse
10	2. Enriches with codebase context (dbt models, gusty SQL, schema docs)
11	3. Records row counts and identifies large tables
12	4. Generates `.astro/warehouse.md` - a version-controllable, team-shareable reference
13	5. Enables instant concept→table lookups without warehouse queries
14
15	## Process
16
17	### Step 1: Read Warehouse Configuration
18
19	```bash
20	cat ~/.astro/agents/warehouse.yml
21	```
22
23	Get the list of databases to discover (e.g., `databases: [HQ, ANALYTICS, RAW]`).
24
25	### Step 2: Search Codebase for Context (Parallel)
26
27	Launch a subagent to find business context in code:
28
29	```
30	Task(
31	subagent_type="Explore",
32	prompt="""
33	Search for data model documentation in the codebase:
34
35	1. dbt models: /models//.yml, */schema.yml
36	- Extract table descriptions, column descriptions
37	- Note primary keys and tests
38
39	2. Gusty/declarative SQL: /dags//*.sql with YAML frontmatter
40	- Parse frontmatter for: description, primary_key, tests
41	- Note schema mappings
42
43	3. AGENTS.md or CLAUDE.md files with data layer documentation
44
45	Return a mapping of:
46	table_name -> {description, primary_key, important_columns, layer}
47	"""
48	)
49	```
50
51	### Step 3: Parallel Warehouse Discovery
52
53	Launch one subagent per database using the Task tool:
54
55	```
56	For each database in configured_databases:
57	Task(
58	subagent_type="general-purpose",
59	prompt="""
60	Discover all metadata for database {DATABASE}.
61
62	Use the CLI to run SQL queries:
63	# Scripts are relative to ../analyzing-data/
64	uv run scripts/cli.py exec "df = run_sql('...')"
65	uv run scripts/cli.py exec "print(df)"
66
67	1. Query schemas:
68	SELECT SCHEMA_NAME FROM {DATABASE}.INFORMATION_SCHEMA.SCHEMATA
69
70	2. Query tables with row counts:
71	SELECT TABLE_SCHEMA, TABLE_NAME, ROW_COUNT, COMMENT
72	FROM {DATABASE}.INFORMATION_SCHEMA.TABLES
73	ORDER BY TABLE_SCHEMA, TABLE_NAME
74
75	3. For important schemas (MODEL_, METRICS_, MART_*), query columns:
76	SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, COMMENT
77	FROM {DATABASE}.INFORMATION_SCHEMA.COLUMNS
78	WHERE TABLE_SCHEMA = 'X'
79
80	Return a structured summary:
81	- Database name
82	- List of schemas with table counts
83	- For each table: name, row_count, key columns
84	- Flag any tables with >100M rows as "large"
85	"""
86	)
87	```
88
89	Run all subagents in parallel (single message with multiple Task calls).
90
91	### Step 4: Discover Categorical Value Families
92
93	For key categorical columns (like OPERATOR, STATUS, TYPE, FEATURE), discover value families:
94
95	```bash
96	uv run cli.py exec "df = run_sql('''
97	SELECT DISTINCT column_name, COUNT(*) as occurrences
98	FROM table
99	WHERE column_name IS NOT NULL
100	GROUP BY column_name
101	ORDER BY occurrences DESC
102	LIMIT 50
103	''')"
104	uv run cli.py exec "print(df)"
105	```
106
107	Group related values into families by common prefix/suffix (e.g., `Export*` for ExportCSV, ExportJSON, ExportParquet).
108
109	### Step 5: Merge Results
110
111	Combine warehouse metadata + codebase context:
112
113	1. Quick Reference table - concept → table mappings (pre-populated from code if found)
114	2. Categorical Columns - value families for key filter columns
115	3. Database sections - one per database
116	4. Schema subsections - tables grouped by schema
117	5. Table details - columns, row counts, descriptions from code, warnings
118
119	### Step 6: Generate warehouse.md
120
121	Write the file to:
122	- `.astro/warehouse.md` (default - project-specific, version-controllable)
123	- `~/.astro/agents/warehouse.md` (if `--global` flag)
124
125	## Output Format
126
127	```markdown
128	# Warehouse Schema
129
130	> Generated by `/astronomer-data:warehouse-init` on {DATE}. Edit freely to add business context.
131
132	## Quick Reference
133
134	\| Concept \| Table \| Key Column \| Date Column \|
135	\|---------\|-------\|------------\|-------------\|
136	\| customers \| HQ.MODEL_ASTRO.ORGANIZATIONS \| ORG_ID \| CREATED_AT \|
137	<!-- Add your concept mappings here -->
138
139	## Categorical Columns
140
141	When filtering on these columns, explore value families first (values often have variants):
142
143	\| Table \| Column \| Value Families \|
144	\|-------\|--------\|----------------\|
145	\| {TABLE} \| {COLUMN} \| `{PREFIX}*` ({VALUE1}, {VALUE2}, ...) \|
146	<!-- Populated by /astronomer-data:warehouse-init from ac

People who liked this also viewed

6 picks

Type

warehouse-init

662installs55forks

View on GitHub

How to install warehouse-init?

astronomer/agents/warehouse-init

$npx -y skills add astronomer/agents --skill warehouse-init

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Initialize Warehouse Schema
2
3	Generate a comprehensive, user-editable schema reference file for the data warehouse.
4
5	Scripts: `../analyzing-data/scripts/` — All CLI commands below are relative to the `analyzing-data` skill's directory. Before running any `scripts/cli.py` command, `cd` to `../analyzing-data/` relative to this file.
6
7	## What This Does
8
9	1. Discovers all databases, schemas, tables, and columns from the warehouse
10	2. Enriches with codebase context (dbt models, gusty SQL, schema docs)
11	3. Records row counts and identifies large tables
12	4. Generates `.astro/warehouse.md` - a version-controllable, team-shareable reference
13	5. Enables instant concept→table lookups without warehouse queries
14
15	## Process
16
17	### Step 1: Read Warehouse Configuration
18
19	```bash
20	cat ~/.astro/agents/warehouse.yml
21	```
22
23	Get the list of databases to discover (e.g., `databases: [HQ, ANALYTICS, RAW]`).
24
25	### Step 2: Search Codebase for Context (Parallel)
26
27	Launch a subagent to find business context in code:
28
29	```
30	Task(
31	subagent_type="Explore",
32	prompt="""
33	Search for data model documentation in the codebase:
34
35	1. dbt models: /models//.yml, */schema.yml
36	- Extract table descriptions, column descriptions
37	- Note primary keys and tests
38
39	2. Gusty/declarative SQL: /dags//*.sql with YAML frontmatter
40	- Parse frontmatter for: description, primary_key, tests
41	- Note schema mappings
42
43	3. AGENTS.md or CLAUDE.md files with data layer documentation
44
45	Return a mapping of:
46	table_name -> {description, primary_key, important_columns, layer}
47	"""
48	)
49	```
50
51	### Step 3: Parallel Warehouse Discovery
52
53	Launch one subagent per database using the Task tool:
54
55	```
56	For each database in configured_databases:
57	Task(
58	subagent_type="general-purpose",
59	prompt="""
60	Discover all metadata for database {DATABASE}.
61
62	Use the CLI to run SQL queries:
63	# Scripts are relative to ../analyzing-data/
64	uv run scripts/cli.py exec "df = run_sql('...')"
65	uv run scripts/cli.py exec "print(df)"
66
67	1. Query schemas:
68	SELECT SCHEMA_NAME FROM {DATABASE}.INFORMATION_SCHEMA.SCHEMATA
69
70	2. Query tables with row counts:
71	SELECT TABLE_SCHEMA, TABLE_NAME, ROW_COUNT, COMMENT
72	FROM {DATABASE}.INFORMATION_SCHEMA.TABLES
73	ORDER BY TABLE_SCHEMA, TABLE_NAME
74
75	3. For important schemas (MODEL_, METRICS_, MART_*), query columns:
76	SELECT TABLE_NAME, COLUMN_NAME, DATA_TYPE, COMMENT
77	FROM {DATABASE}.INFORMATION_SCHEMA.COLUMNS
78	WHERE TABLE_SCHEMA = 'X'
79
80	Return a structured summary:
81	- Database name
82	- List of schemas with table counts
83	- For each table: name, row_count, key columns
84	- Flag any tables with >100M rows as "large"
85	"""
86	)
87	```
88
89	Run all subagents in parallel (single message with multiple Task calls).
90
91	### Step 4: Discover Categorical Value Families
92
93	For key categorical columns (like OPERATOR, STATUS, TYPE, FEATURE), discover value families:
94
95	```bash
96	uv run cli.py exec "df = run_sql('''
97	SELECT DISTINCT column_name, COUNT(*) as occurrences
98	FROM table
99	WHERE column_name IS NOT NULL
100	GROUP BY column_name
101	ORDER BY occurrences DESC
102	LIMIT 50
103	''')"
104	uv run cli.py exec "print(df)"
105	```
106
107	Group related values into families by common prefix/suffix (e.g., `Export*` for ExportCSV, ExportJSON, ExportParquet).
108
109	### Step 5: Merge Results
110
111	Combine warehouse metadata + codebase context:
112
113	1. Quick Reference table - concept → table mappings (pre-populated from code if found)
114	2. Categorical Columns - value families for key filter columns
115	3. Database sections - one per database
116	4. Schema subsections - tables grouped by schema
117	5. Table details - columns, row counts, descriptions from code, warnings
118
119	### Step 6: Generate warehouse.md
120
121	Write the file to:
122	- `.astro/warehouse.md` (default - project-specific, version-controllable)
123	- `~/.astro/agents/warehouse.md` (if `--global` flag)
124
125	## Output Format
126
127	```markdown
128	# Warehouse Schema
129
130	> Generated by `/astronomer-data:warehouse-init` on {DATE}. Edit freely to add business context.
131
132	## Quick Reference
133
134	\| Concept \| Table \| Key Column \| Date Column \|
135	\|---------\|-------\|------------\|-------------\|
136	\| customers \| HQ.MODEL_ASTRO.ORGANIZATIONS \| ORG_ID \| CREATED_AT \|
137	<!-- Add your concept mappings here -->
138
139	## Categorical Columns
140
141	When filtering on these columns, explore value families first (values often have variants):
142
143	\| Table \| Column \| Value Families \|
144	\|-------\|--------\|----------------\|
145	\| {TABLE} \| {COLUMN} \| `{PREFIX}*` ({VALUE1}, {VALUE2}, ...) \|
146	<!-- Populated by /astronomer-data:warehouse-init from ac

People who liked this also viewed

6 picks

Type