profiling-tables

Name: profiling-tables
Author: astronomer

865installs55forks

How to install profiling-tables?

astronomer/agents/profiling-tables

$npx -y skills add astronomer/agents --skill profiling-tables

Installs into the current project.

Summary

Deep-dive data profiling for a specific table. Use when the user asks to profile a table, wants statistics about a dataset, asks about data quality, or needs to understand a table's structure and content. Requires a table name.

Files · 1

View on GitHub

SKILL.md

1	# Data Profile
2
3	Generate a comprehensive profile of a table that a new team member could use to understand the data.
4
5	## Step 1: Basic Metadata
6
7	Query column metadata:
8
9	```sql
10	SELECT COLUMN_NAME, DATA_TYPE, COMMENT
11	FROM <database>.INFORMATION_SCHEMA.COLUMNS
12	WHERE TABLE_SCHEMA = '<schema>' AND TABLE_NAME = '<table>'
13	ORDER BY ORDINAL_POSITION
14	```
15
16	If the table name isn't fully qualified, search INFORMATION_SCHEMA.TABLES to locate it first.
17
18	## Step 2: Size and Shape
19
20	Run via `run_sql`:
21
22	```sql
23	SELECT
24	COUNT(*) as total_rows,
25	COUNT(*) / 1000000.0 as millions_of_rows
26	FROM <table>
27	```
28
29	## Step 3: Column-Level Statistics
30
31	For each column, gather appropriate statistics based on data type:
32
33	### Numeric Columns
34	```sql
35	SELECT
36	MIN(column_name) as min_val,
37	MAX(column_name) as max_val,
38	AVG(column_name) as avg_val,
39	STDDEV(column_name) as std_dev,
40	PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY column_name) as median,
41	SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) as null_count,
42	COUNT(DISTINCT column_name) as distinct_count
43	FROM <table>
44	```
45
46	### String Columns
47	```sql
48	SELECT
49	MIN(LEN(column_name)) as min_length,
50	MAX(LEN(column_name)) as max_length,
51	AVG(LEN(column_name)) as avg_length,
52	SUM(CASE WHEN column_name IS NULL OR column_name = '' THEN 1 ELSE 0 END) as empty_count,
53	COUNT(DISTINCT column_name) as distinct_count
54	FROM <table>
55	```
56
57	### Date/Timestamp Columns
58	```sql
59	SELECT
60	MIN(column_name) as earliest,
61	MAX(column_name) as latest,
62	DATEDIFF('day', MIN(column_name), MAX(column_name)) as date_range_days,
63	SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) as null_count
64	FROM <table>
65	```
66
67	## Step 4: Cardinality Analysis
68
69	For columns that look like categorical/dimension keys:
70
71	```sql
72	SELECT
73	column_name,
74	COUNT(*) as frequency,
75	ROUND(COUNT() 100.0 / SUM(COUNT(*)) OVER(), 2) as percentage
76	FROM <table>
77	GROUP BY column_name
78	ORDER BY frequency DESC
79	LIMIT 20
80	```
81
82	This reveals:
83	- High-cardinality columns (likely IDs or unique values)
84	- Low-cardinality columns (likely categories or status fields)
85	- Skewed distributions (one value dominates)
86
87	## Step 5: Sample Data
88
89	Get representative rows:
90
91	```sql
92	SELECT *
93	FROM <table>
94	LIMIT 10
95	```
96
97	If the table is large and you want variety, sample from different time periods or categories.
98
99	## Step 6: Data Quality Assessment
100
101	Summarize quality across dimensions:
102
103	### Completeness
104	- Which columns have NULLs? What percentage?
105	- Are NULLs expected or problematic?
106
107	### Uniqueness
108	- Does the apparent primary key have duplicates?
109	- Are there unexpected duplicate rows?
110
111	### Freshness
112	- When was data last updated? (MAX of timestamp columns)
113	- Is the update frequency as expected?
114
115	### Validity
116	- Are there values outside expected ranges?
117	- Are there invalid formats (dates, emails, etc.)?
118	- Are there orphaned foreign keys?
119
120	### Consistency
121	- Do related columns make sense together?
122	- Are there logical contradictions?
123
124	## Step 7: Output Summary
125
126	Provide a structured profile:
127
128	### Overview
129	2-3 sentences describing what this table contains, who uses it, and how fresh it is.
130
131	### Schema
132	\| Column \| Type \| Nulls% \| Distinct \| Description \|
133	\|--------\|------\|--------\|----------\|-------------\|
134	\| ... \| ... \| ... \| ... \| ... \|
135
136	### Key Statistics
137	- Row count: X
138	- Date range: Y to Z
139	- Last updated: timestamp
140
141	### Data Quality Score
142	- Completeness: X/10
143	- Uniqueness: X/10
144	- Freshness: X/10
145	- Overall: X/10
146
147	### Potential Issues
148	List any data quality concerns discovered.
149
150	### Recommended Queries
151	3-5 useful queries for common questions about this data.

People who liked this also viewed

6 picks

Type

profiling-tables

865installs55forks

View on GitHub

How to install profiling-tables?

astronomer/agents/profiling-tables

$npx -y skills add astronomer/agents --skill profiling-tables

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Profile
2
3	Generate a comprehensive profile of a table that a new team member could use to understand the data.
4
5	## Step 1: Basic Metadata
6
7	Query column metadata:
8
9	```sql
10	SELECT COLUMN_NAME, DATA_TYPE, COMMENT
11	FROM <database>.INFORMATION_SCHEMA.COLUMNS
12	WHERE TABLE_SCHEMA = '<schema>' AND TABLE_NAME = '<table>'
13	ORDER BY ORDINAL_POSITION
14	```
15
16	If the table name isn't fully qualified, search INFORMATION_SCHEMA.TABLES to locate it first.
17
18	## Step 2: Size and Shape
19
20	Run via `run_sql`:
21
22	```sql
23	SELECT
24	COUNT(*) as total_rows,
25	COUNT(*) / 1000000.0 as millions_of_rows
26	FROM <table>
27	```
28
29	## Step 3: Column-Level Statistics
30
31	For each column, gather appropriate statistics based on data type:
32
33	### Numeric Columns
34	```sql
35	SELECT
36	MIN(column_name) as min_val,
37	MAX(column_name) as max_val,
38	AVG(column_name) as avg_val,
39	STDDEV(column_name) as std_dev,
40	PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY column_name) as median,
41	SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) as null_count,
42	COUNT(DISTINCT column_name) as distinct_count
43	FROM <table>
44	```
45
46	### String Columns
47	```sql
48	SELECT
49	MIN(LEN(column_name)) as min_length,
50	MAX(LEN(column_name)) as max_length,
51	AVG(LEN(column_name)) as avg_length,
52	SUM(CASE WHEN column_name IS NULL OR column_name = '' THEN 1 ELSE 0 END) as empty_count,
53	COUNT(DISTINCT column_name) as distinct_count
54	FROM <table>
55	```
56
57	### Date/Timestamp Columns
58	```sql
59	SELECT
60	MIN(column_name) as earliest,
61	MAX(column_name) as latest,
62	DATEDIFF('day', MIN(column_name), MAX(column_name)) as date_range_days,
63	SUM(CASE WHEN column_name IS NULL THEN 1 ELSE 0 END) as null_count
64	FROM <table>
65	```
66
67	## Step 4: Cardinality Analysis
68
69	For columns that look like categorical/dimension keys:
70
71	```sql
72	SELECT
73	column_name,
74	COUNT(*) as frequency,
75	ROUND(COUNT() 100.0 / SUM(COUNT(*)) OVER(), 2) as percentage
76	FROM <table>
77	GROUP BY column_name
78	ORDER BY frequency DESC
79	LIMIT 20
80	```
81
82	This reveals:
83	- High-cardinality columns (likely IDs or unique values)
84	- Low-cardinality columns (likely categories or status fields)
85	- Skewed distributions (one value dominates)
86
87	## Step 5: Sample Data
88
89	Get representative rows:
90
91	```sql
92	SELECT *
93	FROM <table>
94	LIMIT 10
95	```
96
97	If the table is large and you want variety, sample from different time periods or categories.
98
99	## Step 6: Data Quality Assessment
100
101	Summarize quality across dimensions:
102
103	### Completeness
104	- Which columns have NULLs? What percentage?
105	- Are NULLs expected or problematic?
106
107	### Uniqueness
108	- Does the apparent primary key have duplicates?
109	- Are there unexpected duplicate rows?
110
111	### Freshness
112	- When was data last updated? (MAX of timestamp columns)
113	- Is the update frequency as expected?
114
115	### Validity
116	- Are there values outside expected ranges?
117	- Are there invalid formats (dates, emails, etc.)?
118	- Are there orphaned foreign keys?
119
120	### Consistency
121	- Do related columns make sense together?
122	- Are there logical contradictions?
123
124	## Step 7: Output Summary
125
126	Provide a structured profile:
127
128	### Overview
129	2-3 sentences describing what this table contains, who uses it, and how fresh it is.
130
131	### Schema
132	\| Column \| Type \| Nulls% \| Distinct \| Description \|
133	\|--------\|------\|--------\|----------\|-------------\|
134	\| ... \| ... \| ... \| ... \| ... \|
135
136	### Key Statistics
137	- Row count: X
138	- Date range: Y to Z
139	- Last updated: timestamp
140
141	### Data Quality Score
142	- Completeness: X/10
143	- Uniqueness: X/10
144	- Freshness: X/10
145	- Overall: X/10
146
147	### Potential Issues
148	List any data quality concerns discovered.
149
150	### Recommended Queries
151	3-5 useful queries for common questions about this data.

People who liked this also viewed

6 picks

Type