dataset-onboard

Name: dataset-onboard
Author: zakelfassi

17stars0forks

How to install dataset-onboard?

zakelfassi/skills-driven-development/dataset-onboard

$npx -y skills add zakelfassi/skills-driven-development --skill dataset-onboard

Installs into the current project.

Summary

Onboard a new raw dataset into the data platform — sniff the schema, generate a profiling notebook, create the ingestion job, and add an entry to the data dictionary. Use when adding a new data source, when a partner delivers a new CSV/Parquet drop, or when asked to "onboard the

Files · 1

View on GitHub

SKILL.md

1	# Dataset Onboard
2
3	Bring a new raw dataset into the platform with schema documentation, profiling, and an ingestion job.
4
5	## Inputs
6	- Dataset name (snake_case, e.g., `customer_events`)
7	- Source format (`csv`, `parquet`, `json`, `avro`)
8	- Source path or URI (S3, GCS, local mount, API endpoint)
9	- Expected frequency (`daily`, `weekly`, `on-demand`)
10	- Owner team and contact
11
12	## Steps
13
14	1. Schema sniff
15	```python
16	import pandas as pd
17	df = pd.read_csv("{source_path}", nrows=1000) # or read_parquet, etc.
18	print(df.dtypes)
19	print(df.describe(include="all"))
20	print(df.isnull().sum() / len(df)) # null rates
21	```
22	Document:
23	- Column names, inferred types, null rates, example values
24	- Detected anomalies (mixed types, encoding issues, unexpected nulls)
25
26	2. Create the data dictionary entry
27	Edit `docs/data-dictionary/{dataset_name}.md`:
28	```markdown
29	# {DatasetName}
30
31	Owner: {team} Contact: {email}
32	Source: {uri} Frequency: {frequency}
33
34	\| Column \| Type \| Nullable \| Description \|
35	\|--------\|------\|----------\|-------------\|
36	\| ... \| ... \| ... \| ... \|
37	```
38
39	3. Generate the profiling notebook
40	```bash
41	cp templates/profiling-notebook.ipynb \
42	notebooks/profiling/{dataset_name}_profile.ipynb
43	```
44	Edit the notebook to use the correct source path and column list.
45	Run it to confirm it completes without errors.
46
47	4. Create the ingestion job
48	```
49	pipelines/ingestion/{dataset_name}/
50	├── ingest.py # main ingestion script
51	├── schema.py # column definitions and type coercion
52	├── config.yaml # source path, schedule, destination table
53	└── tests/
54	└── test_ingest.py # unit test with a small fixture file
55	```
56	The ingestion script must be idempotent (re-running on the same input produces the same output; no duplicate rows).
57
58	5. Register in the scheduler
59	Add a DAG entry in `dags/{dataset_name}_ingest.py` (Airflow) or a flow in `flows/` (Prefect).
60	Set the schedule to match `{frequency}`.
61
62	6. Test the ingestion
63	```bash
64	python -m pytest pipelines/ingestion/{dataset_name}/tests/ -v
65	python pipelines/ingestion/{dataset_name}/ingest.py --dry-run
66	```
67
68	7. Run a data quality gate (invoke `data-quality-gate` skill)
69	Add at minimum: null checks for required columns, row-count sanity check.
70
71	## Conventions
72	- Raw datasets land in the `raw/` schema/layer; never write to `staging/` or `marts/` from an ingestion job
73	- All ingestion jobs accept `--dry-run` and `--date` flags
74	- Dataset names are snake_case; tables follow the same naming
75	- Profiling notebooks live in `notebooks/profiling/`; they are committed
76
77	## Edge Cases
78	- Malformed source file: Log the error with row number, skip the bad row, emit a `data_quality_alert` metric. Never silently swallow rows.
79	- Schema drift (columns added/removed): The ingestion job must compare the inbound schema to `schema.py` and fail fast on unexpected changes, rather than silently loading partial data.
80	- Large datasets (>1GB): Use chunked reads (`chunksize` in pandas, or native Parquet partitioning); test with a 10k-row sample first.
81	- API source with rate limits: Add exponential back-off and a `--resume-from` flag that uses a checkpoint file.

People who liked this also viewed

6 picks

Type

dataset-onboard

17stars0forks

View on GitHub

How to install dataset-onboard?

zakelfassi/skills-driven-development/dataset-onboard

$npx -y skills add zakelfassi/skills-driven-development --skill dataset-onboard

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Dataset Onboard
2
3	Bring a new raw dataset into the platform with schema documentation, profiling, and an ingestion job.
4
5	## Inputs
6	- Dataset name (snake_case, e.g., `customer_events`)
7	- Source format (`csv`, `parquet`, `json`, `avro`)
8	- Source path or URI (S3, GCS, local mount, API endpoint)
9	- Expected frequency (`daily`, `weekly`, `on-demand`)
10	- Owner team and contact
11
12	## Steps
13
14	1. Schema sniff
15	```python
16	import pandas as pd
17	df = pd.read_csv("{source_path}", nrows=1000) # or read_parquet, etc.
18	print(df.dtypes)
19	print(df.describe(include="all"))
20	print(df.isnull().sum() / len(df)) # null rates
21	```
22	Document:
23	- Column names, inferred types, null rates, example values
24	- Detected anomalies (mixed types, encoding issues, unexpected nulls)
25
26	2. Create the data dictionary entry
27	Edit `docs/data-dictionary/{dataset_name}.md`:
28	```markdown
29	# {DatasetName}
30
31	Owner: {team} Contact: {email}
32	Source: {uri} Frequency: {frequency}
33
34	\| Column \| Type \| Nullable \| Description \|
35	\|--------\|------\|----------\|-------------\|
36	\| ... \| ... \| ... \| ... \|
37	```
38
39	3. Generate the profiling notebook
40	```bash
41	cp templates/profiling-notebook.ipynb \
42	notebooks/profiling/{dataset_name}_profile.ipynb
43	```
44	Edit the notebook to use the correct source path and column list.
45	Run it to confirm it completes without errors.
46
47	4. Create the ingestion job
48	```
49	pipelines/ingestion/{dataset_name}/
50	├── ingest.py # main ingestion script
51	├── schema.py # column definitions and type coercion
52	├── config.yaml # source path, schedule, destination table
53	└── tests/
54	└── test_ingest.py # unit test with a small fixture file
55	```
56	The ingestion script must be idempotent (re-running on the same input produces the same output; no duplicate rows).
57
58	5. Register in the scheduler
59	Add a DAG entry in `dags/{dataset_name}_ingest.py` (Airflow) or a flow in `flows/` (Prefect).
60	Set the schedule to match `{frequency}`.
61
62	6. Test the ingestion
63	```bash
64	python -m pytest pipelines/ingestion/{dataset_name}/tests/ -v
65	python pipelines/ingestion/{dataset_name}/ingest.py --dry-run
66	```
67
68	7. Run a data quality gate (invoke `data-quality-gate` skill)
69	Add at minimum: null checks for required columns, row-count sanity check.
70
71	## Conventions
72	- Raw datasets land in the `raw/` schema/layer; never write to `staging/` or `marts/` from an ingestion job
73	- All ingestion jobs accept `--dry-run` and `--date` flags
74	- Dataset names are snake_case; tables follow the same naming
75	- Profiling notebooks live in `notebooks/profiling/`; they are committed
76
77	## Edge Cases
78	- Malformed source file: Log the error with row number, skip the bad row, emit a `data_quality_alert` metric. Never silently swallow rows.
79	- Schema drift (columns added/removed): The ingestion job must compare the inbound schema to `schema.py` and fail fast on unexpected changes, rather than silently loading partial data.
80	- Large datasets (>1GB): Use chunked reads (`chunksize` in pandas, or native Parquet partitioning); test with a 10k-row sample first.
81	- API source with rate limits: Add exponential back-off and a `--resume-from` flag that uses a checkpoint file.

People who liked this also viewed

6 picks

Type