pipeline-planning-and-task-breakdown

Name: pipeline-planning-and-task-breakdown
Author: vaquarkhan

19stars4forks

How to install pipeline-planning-and-task-breakdown?

vaquarkhan/data-engineering-agent-skills/pipeline-planning-and-task-breakdown

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill pipeline-planning-and-task-breakdown

Installs into the current project.

Summary

Breaks approved data specifications into safe, verifiable implementation tasks. Use when a data project spans multiple steps, systems, or files and needs dependency-aware sequencing.

Files · 1

View on GitHub

SKILL.md

1	# Pipeline Planning And Task Breakdown
2
3	## Overview
4
5	Turn a validated data specification into a sequence of small tasks that can be implemented and verified independently. A good plan protects the team from wide, risky pipeline edits.
6
7	## When to Use
8
9	- multi-step ingestion or modeling work
10	- changes involving orchestration, infrastructure, and transformations
11	- work that needs coordination across contracts, checks, and deployment
12
13	Do not use this when the task is a single isolated change with obvious verification.
14
15	## Workflow
16
17	1. Map the affected layers.
18	Typical layers include:
19	- source connections
20	- raw landing
21	- staging transformations
22	- business models
23	- orchestration
24	- tests and checks
25	- documentation and runbooks
26
27	2. Order work by dependency.
28	Good sequence:
29	- contracts
30	- scaffolding
31	- ingestion
32	- transformations
33	- checks
34	- orchestration
35	- rollout and recovery notes
36
37	3. Keep tasks small.
38	Each task should:
39	- have one clear outcome
40	- touch a limited set of files
41	- define how it will be verified
42	- avoid mixing unrelated concerns
43
44	4. Call out risk points.
45	Always identify:
46	- schema evolution risk
47	- backfill risk
48	- cost or performance risk
49	- downstream compatibility risk
50
51	5. Define evidence for every task.
52	Examples:
53	- test results
54	- contract validation output
55	- successful dry run
56	- row count reconciliation
57	- lineage or docs update
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "It is faster to change everything in one pass." \| Wide changes increase blast radius and make rollback harder. \|
64	\| "We can skip task boundaries because the repo is small." \| Data dependencies, not repo size, create risk. \|
65	\| "The orchestrator wiring can wait until the end." \| Scheduling and retry behavior often shape the implementation. \|
66
67	## Red Flags
68
69	- a single task spans ingestion, modeling, orchestration, and rollout
70	- verification says only "run the pipeline"
71	- rollback or compatibility concerns are not mentioned
72	- the plan does not separate build work from publish work
73
74	## Verification
75
76	- [ ] Tasks are ordered by dependency
77	- [ ] Each task has acceptance criteria and evidence
78	- [ ] Risks and rollback concerns are called out
79	- [ ] Checks and documentation are part of the plan, not afterthoughts

People who liked this also viewed

6 picks

Type

pipeline-planning-and-task-breakdown

19stars4forks

View on GitHub

How to install pipeline-planning-and-task-breakdown?

vaquarkhan/data-engineering-agent-skills/pipeline-planning-and-task-breakdown

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill pipeline-planning-and-task-breakdown

Installs into the current project.

Summary

Breaks approved data specifications into safe, verifiable implementation tasks. Use when a data project spans multiple steps, systems, or files and needs dependency-aware sequencing.

Files · 1

View on GitHub

SKILL.md

1	# Pipeline Planning And Task Breakdown
2
3	## Overview
4
5	Turn a validated data specification into a sequence of small tasks that can be implemented and verified independently. A good plan protects the team from wide, risky pipeline edits.
6
7	## When to Use
8
9	- multi-step ingestion or modeling work
10	- changes involving orchestration, infrastructure, and transformations
11	- work that needs coordination across contracts, checks, and deployment
12
13	Do not use this when the task is a single isolated change with obvious verification.
14
15	## Workflow
16
17	1. Map the affected layers.
18	Typical layers include:
19	- source connections
20	- raw landing
21	- staging transformations
22	- business models
23	- orchestration
24	- tests and checks
25	- documentation and runbooks
26
27	2. Order work by dependency.
28	Good sequence:
29	- contracts
30	- scaffolding
31	- ingestion
32	- transformations
33	- checks
34	- orchestration
35	- rollout and recovery notes
36
37	3. Keep tasks small.
38	Each task should:
39	- have one clear outcome
40	- touch a limited set of files
41	- define how it will be verified
42	- avoid mixing unrelated concerns
43
44	4. Call out risk points.
45	Always identify:
46	- schema evolution risk
47	- backfill risk
48	- cost or performance risk
49	- downstream compatibility risk
50
51	5. Define evidence for every task.
52	Examples:
53	- test results
54	- contract validation output
55	- successful dry run
56	- row count reconciliation
57	- lineage or docs update
58
59	## Common Rationalizations
60
61	\| Rationalization \| Reality \|
62	\| --- \| --- \|
63	\| "It is faster to change everything in one pass." \| Wide changes increase blast radius and make rollback harder. \|
64	\| "We can skip task boundaries because the repo is small." \| Data dependencies, not repo size, create risk. \|
65	\| "The orchestrator wiring can wait until the end." \| Scheduling and retry behavior often shape the implementation. \|
66
67	## Red Flags
68
69	- a single task spans ingestion, modeling, orchestration, and rollout
70	- verification says only "run the pipeline"
71	- rollback or compatibility concerns are not mentioned
72	- the plan does not separate build work from publish work
73
74	## Verification
75
76	- [ ] Tasks are ordered by dependency
77	- [ ] Each task has acceptance criteria and evidence
78	- [ ] Risks and rollback concerns are called out
79	- [ ] Checks and documentation are part of the plan, not afterthoughts

People who liked this also viewed

6 picks

Type