bigquery-and-dataform-platform-engineering

Name: bigquery-and-dataform-platform-engineering
Author: vaquarkhan

19stars4forks

How to install bigquery-and-dataform-platform-engineering?

vaquarkhan/data-engineering-agent-skills/bigquery-and-dataform-platform-engineering

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill bigquery-and-dataform-platform-engineering

Installs into the current project.

Summary

Guides agents through BigQuery- and Dataform-centered data engineering workflows. Use when designing BigQuery physical models, ingestion boundaries, Dataform transformation workflows, slot or cost controls, and platform decisions across BigQuery, Dataflow, Dataproc, and GCP orche

Files · 1

View on GitHub

SKILL.md

1	# BigQuery And Dataform Platform Engineering
2
3	## Overview
4
5	Use this skill when `BigQuery` is the center of gravity for analytics engineering and data delivery on `GCP`. It helps agents decide what should run in `BigQuery`, what belongs in `Dataform`, and when the workflow should move to `Dataflow`, `Dataproc`, or external orchestration.
6
7	## When to Use
8
9	- designing or reviewing `BigQuery` physical models
10	- deciding between `Dataform`, `dbt`, `Dataflow`, or `Dataproc` responsibilities
11	- tuning partitioning, clustering, slots, and cost behavior
12	- defining ingestion and transformation boundaries on `GCP`
13	- building platform-native analytics workflows around `BigQuery`
14
15	Do not treat `BigQuery` as a universal default for every preprocessing and orchestration need.
16
17	## Workflow
18
19	1. Define the workload boundary.
20	Clarify:
21	- landing pattern
22	- transformation complexity
23	- latency requirements
24	- governance and regional constraints
25	- cost sensitivity
26
27	2. Design `BigQuery` physical layout intentionally.
28	Cover:
29	- partitioning
30	- clustering
31	- dataset boundaries
32	- publish layers
33	- data retention and serving expectations
34
35	3. Choose the transformation surface.
36	Consider:
37	- `Dataform` for warehouse-native SQL transformation workflows
38	- `dbt` when the team already standardizes there
39	- `Dataflow` or `Dataproc` when preprocessing or runtime requirements exceed warehouse-native fit
40
41	4. Define orchestration and operations.
42	Include:
43	- where orchestration runs
44	- slot and concurrency behavior
45	- failure and rerun expectations
46	- validation gates before publish
47
48	5. Validate cost and governance behavior.
49	Require:
50	- slot or query cost awareness
51	- service-account and secret controls
52	- policy tags or governance metadata where needed
53	- publish-readiness evidence
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "Everything on GCP should run in BigQuery." \| Some preprocessing, streaming, or protocol-heavy work belongs in `Dataflow`, `Dataproc`, or upstream services. \|
60	\| "Dataform is just a SQL wrapper." \| It changes how transformation workflows, dependencies, testing, and deployment are managed. \|
61	\| "Partitioning and clustering can be tuned later." \| Poor physical design often becomes a long-term cost and performance tax. \|
62
63	## Red Flags
64
65	- BigQuery physical design is implicit or copied from defaults
66	- transformation boundaries between `Dataform`, `Dataflow`, and `Dataproc` are unclear
67	- cost or slot behavior is not considered during model design
68	- publish and validation behavior is undefined for warehouse-native pipelines
69	- governance metadata and access are treated as separate cleanup work
70
71	## Verification
72
73	- [ ] BigQuery physical design choices are explicit and workload-aware
74	- [ ] Dataform, BigQuery, and other GCP services have clear boundaries
75	- [ ] Cost, slots, and concurrency implications are documented
76	- [ ] Governance and access controls are accounted for
77	- [ ] Publish and validation behavior are explicit before release

People who liked this also viewed

6 picks

Type

bigquery-and-dataform-platform-engineering

19stars4forks

View on GitHub

How to install bigquery-and-dataform-platform-engineering?

vaquarkhan/data-engineering-agent-skills/bigquery-and-dataform-platform-engineering

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill bigquery-and-dataform-platform-engineering

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# BigQuery And Dataform Platform Engineering
2
3	## Overview
4
5	Use this skill when `BigQuery` is the center of gravity for analytics engineering and data delivery on `GCP`. It helps agents decide what should run in `BigQuery`, what belongs in `Dataform`, and when the workflow should move to `Dataflow`, `Dataproc`, or external orchestration.
6
7	## When to Use
8
9	- designing or reviewing `BigQuery` physical models
10	- deciding between `Dataform`, `dbt`, `Dataflow`, or `Dataproc` responsibilities
11	- tuning partitioning, clustering, slots, and cost behavior
12	- defining ingestion and transformation boundaries on `GCP`
13	- building platform-native analytics workflows around `BigQuery`
14
15	Do not treat `BigQuery` as a universal default for every preprocessing and orchestration need.
16
17	## Workflow
18
19	1. Define the workload boundary.
20	Clarify:
21	- landing pattern
22	- transformation complexity
23	- latency requirements
24	- governance and regional constraints
25	- cost sensitivity
26
27	2. Design `BigQuery` physical layout intentionally.
28	Cover:
29	- partitioning
30	- clustering
31	- dataset boundaries
32	- publish layers
33	- data retention and serving expectations
34
35	3. Choose the transformation surface.
36	Consider:
37	- `Dataform` for warehouse-native SQL transformation workflows
38	- `dbt` when the team already standardizes there
39	- `Dataflow` or `Dataproc` when preprocessing or runtime requirements exceed warehouse-native fit
40
41	4. Define orchestration and operations.
42	Include:
43	- where orchestration runs
44	- slot and concurrency behavior
45	- failure and rerun expectations
46	- validation gates before publish
47
48	5. Validate cost and governance behavior.
49	Require:
50	- slot or query cost awareness
51	- service-account and secret controls
52	- policy tags or governance metadata where needed
53	- publish-readiness evidence
54
55	## Common Rationalizations
56
57	\| Rationalization \| Reality \|
58	\| --- \| --- \|
59	\| "Everything on GCP should run in BigQuery." \| Some preprocessing, streaming, or protocol-heavy work belongs in `Dataflow`, `Dataproc`, or upstream services. \|
60	\| "Dataform is just a SQL wrapper." \| It changes how transformation workflows, dependencies, testing, and deployment are managed. \|
61	\| "Partitioning and clustering can be tuned later." \| Poor physical design often becomes a long-term cost and performance tax. \|
62
63	## Red Flags
64
65	- BigQuery physical design is implicit or copied from defaults
66	- transformation boundaries between `Dataform`, `Dataflow`, and `Dataproc` are unclear
67	- cost or slot behavior is not considered during model design
68	- publish and validation behavior is undefined for warehouse-native pipelines
69	- governance metadata and access are treated as separate cleanup work
70
71	## Verification
72
73	- [ ] BigQuery physical design choices are explicit and workload-aware
74	- [ ] Dataform, BigQuery, and other GCP services have clear boundaries
75	- [ ] Cost, slots, and concurrency implications are documented
76	- [ ] Governance and access controls are accounted for
77	- [ ] Publish and validation behavior are explicit before release

People who liked this also viewed

6 picks

Type