duckdb-local-analytics-and-dev

Name: duckdb-local-analytics-and-dev
Author: vaquarkhan

19stars4forks

How to install duckdb-local-analytics-and-dev?

vaquarkhan/data-engineering-agent-skills/duckdb-local-analytics-and-dev

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill duckdb-local-analytics-and-dev

Installs into the current project.

Summary

Guides agents through DuckDB-based local analytics and development workflows. Use when prototyping models locally, validating transformations, reproducing data issues quickly, or building lightweight analytical tooling without a full warehouse.

Files · 1

View on GitHub

SKILL.md

1	# DuckDB Local Analytics And Dev
2
3	## Overview
4
5	Use this skill when `DuckDB` is the fastest path to local analytical iteration. It helps agents build reproducible local workflows that accelerate development without confusing prototype convenience for production architecture.
6
7	## When to Use
8
9	- prototyping data models and transformations locally before deploying to a warehouse
10	- reproducing production data issues with sample datasets
11	- running analytical queries during development without remote infrastructure
12	- building lightweight CLI tools, validators, or test harnesses
13	- validating dbt models locally with `dbt-duckdb` adapter
14	- creating proof-of-concept demonstrations with embedded analytics
15
16	Do not use this when the workload requires production durability, concurrent access, or distributed processing. DuckDB is a development and prototyping accelerator, not a production warehouse replacement.
17
18	## Workflow
19
20	1. Define the purpose and scope of the local workflow.
21	Include:
22	- what question or validation is this workflow answering?
23	- what sample data is needed and where does it come from?
24	- is this a one-time investigation or a repeatable development workflow?
25	- what is the promotion path to production if the prototype succeeds?
26
27	2. Set up reproducible data inputs.
28	- use sample files (CSV, Parquet, JSON) checked into the repository or downloaded by script
29	- document how sample data was generated or extracted
30	- keep sample sizes representative but small enough for fast iteration
31	- use DuckDB's ability to read Parquet, CSV, and JSON directly without import steps
32	- for sensitive data: use anonymized or synthetic samples only
33
34	3. Write transformations that map cleanly to production equivalents.
35	- use standard SQL that translates to the target warehouse dialect
36	- avoid DuckDB-specific functions unless the workflow stays local permanently
37	- structure queries in the same layered pattern (staging → intermediate → marts) as production
38	- when using `dbt-duckdb`: use the same model structure and tests as the production adapter
39	- document which DuckDB-specific features would need replacement in production
40
41	4. Validate results locally with assertions and contract checks.
42	- run row count checks, null assertions, and key uniqueness tests
43	- compare output against expected results or golden files
44	- use DuckDB's SUMMARIZE and descriptive statistics for quick sanity checks
45	- integrate with the project's validation framework (Great Expectations, Cuallee, or custom)
46
47	5. Document the production promotion path.
48	- what must change before this prototype runs in production?
49	- list DuckDB-specific assumptions: single-node, in-process, file-based storage
50	- define the target platform (Snowflake, BigQuery, Redshift, Spark) and dialect differences
51	- identify features that need distributed execution (large joins, window functions at scale)
52	- make promotion a conscious decision, not an accident
53
54	6. Keep the local workflow maintainable.
55	- include a `Makefile` or script that runs the full local workflow from scratch
56	- pin DuckDB version in requirements to prevent drift
57	- clean up temporary databases between runs
58	- document expected execution time so developers know what's normal
59	- retire local workflows that no longer serve a purpose
60
61	## Common Rationalizations
62
63	\| Rationalization \| Reality \|
64	\| --- \| --- \|
65	\| "DuckDB is so fast, we can just use it in production." \| DuckDB is single-process and file-based. Production workloads need concurrency, durability, and operational tooling that DuckDB does not provide. \|
66	\| "The local prototype is basically the same as production." \| Local prototypes skip authentication, network, concurrency, and scale concerns. The gap between local and production must be explicitly documented. \|
67	\| "We don't need sample data management — just point at production files." \| Pointing at production data from local machines creates security and size problems. Curated samples are safer and faster. \|
68	\| "It works locally so it will work in the warehouse." \| DuckDB SQL is largely standard but not identical to Snowflake, BigQuery, or Redshift dialects. Promotion requires testing on the real target. \|
69
70	## Red Flags
71
72	- local DuckDB workflow is treated as production without a promotion plan
73	- sample data includes real PII or production secrets
74	- DuckDB-specific functions are used without documenting production equivalents
75	- no Makefile or script to reproduce the workflow from scratch
76	- local workflow runs against full production-scale files on a laptop
77	- prototype models are deployed to production without validation on the target platform
78	- temporary Du

People who liked this also viewed

6 picks

Type

duckdb-local-analytics-and-dev

19stars4forks

View on GitHub

How to install duckdb-local-analytics-and-dev?

vaquarkhan/data-engineering-agent-skills/duckdb-local-analytics-and-dev

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill duckdb-local-analytics-and-dev

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# DuckDB Local Analytics And Dev
2
3	## Overview
4
5	Use this skill when `DuckDB` is the fastest path to local analytical iteration. It helps agents build reproducible local workflows that accelerate development without confusing prototype convenience for production architecture.
6
7	## When to Use
8
9	- prototyping data models and transformations locally before deploying to a warehouse
10	- reproducing production data issues with sample datasets
11	- running analytical queries during development without remote infrastructure
12	- building lightweight CLI tools, validators, or test harnesses
13	- validating dbt models locally with `dbt-duckdb` adapter
14	- creating proof-of-concept demonstrations with embedded analytics
15
16	Do not use this when the workload requires production durability, concurrent access, or distributed processing. DuckDB is a development and prototyping accelerator, not a production warehouse replacement.
17
18	## Workflow
19
20	1. Define the purpose and scope of the local workflow.
21	Include:
22	- what question or validation is this workflow answering?
23	- what sample data is needed and where does it come from?
24	- is this a one-time investigation or a repeatable development workflow?
25	- what is the promotion path to production if the prototype succeeds?
26
27	2. Set up reproducible data inputs.
28	- use sample files (CSV, Parquet, JSON) checked into the repository or downloaded by script
29	- document how sample data was generated or extracted
30	- keep sample sizes representative but small enough for fast iteration
31	- use DuckDB's ability to read Parquet, CSV, and JSON directly without import steps
32	- for sensitive data: use anonymized or synthetic samples only
33
34	3. Write transformations that map cleanly to production equivalents.
35	- use standard SQL that translates to the target warehouse dialect
36	- avoid DuckDB-specific functions unless the workflow stays local permanently
37	- structure queries in the same layered pattern (staging → intermediate → marts) as production
38	- when using `dbt-duckdb`: use the same model structure and tests as the production adapter
39	- document which DuckDB-specific features would need replacement in production
40
41	4. Validate results locally with assertions and contract checks.
42	- run row count checks, null assertions, and key uniqueness tests
43	- compare output against expected results or golden files
44	- use DuckDB's SUMMARIZE and descriptive statistics for quick sanity checks
45	- integrate with the project's validation framework (Great Expectations, Cuallee, or custom)
46
47	5. Document the production promotion path.
48	- what must change before this prototype runs in production?
49	- list DuckDB-specific assumptions: single-node, in-process, file-based storage
50	- define the target platform (Snowflake, BigQuery, Redshift, Spark) and dialect differences
51	- identify features that need distributed execution (large joins, window functions at scale)
52	- make promotion a conscious decision, not an accident
53
54	6. Keep the local workflow maintainable.
55	- include a `Makefile` or script that runs the full local workflow from scratch
56	- pin DuckDB version in requirements to prevent drift
57	- clean up temporary databases between runs
58	- document expected execution time so developers know what's normal
59	- retire local workflows that no longer serve a purpose
60
61	## Common Rationalizations
62
63	\| Rationalization \| Reality \|
64	\| --- \| --- \|
65	\| "DuckDB is so fast, we can just use it in production." \| DuckDB is single-process and file-based. Production workloads need concurrency, durability, and operational tooling that DuckDB does not provide. \|
66	\| "The local prototype is basically the same as production." \| Local prototypes skip authentication, network, concurrency, and scale concerns. The gap between local and production must be explicitly documented. \|
67	\| "We don't need sample data management — just point at production files." \| Pointing at production data from local machines creates security and size problems. Curated samples are safer and faster. \|
68	\| "It works locally so it will work in the warehouse." \| DuckDB SQL is largely standard but not identical to Snowflake, BigQuery, or Redshift dialects. Promotion requires testing on the real target. \|
69
70	## Red Flags
71
72	- local DuckDB workflow is treated as production without a promotion plan
73	- sample data includes real PII or production secrets
74	- DuckDB-specific functions are used without documenting production equivalents
75	- no Makefile or script to reproduce the workflow from scratch
76	- local workflow runs against full production-scale files on a laptop
77	- prototype models are deployed to production without validation on the target platform
78	- temporary Du

People who liked this also viewed

6 picks

Type