orchestration-and-backfills

Name: orchestration-and-backfills
Author: vaquarkhan

19stars4forks

How to install orchestration-and-backfills?

vaquarkhan/data-engineering-agent-skills/orchestration-and-backfills

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill orchestration-and-backfills

Installs into the current project.

Summary

Designs scheduling, reruns, and backfills safely for data systems. Use when changing orchestration, retries, dependency timing, historical reprocessing, or publish sequencing.

Files · 1

View on GitHub

SKILL.md

1	# Orchestration And Backfills
2
3	## Overview
4
5	Reliable data systems are shaped as much by rerun behavior as by transformation logic. This skill ensures schedules, retries, and backfills are deliberate, safe, and reversible.
6
7	## When to Use
8
9	- adding or changing a scheduled pipeline
10	- designing retries or failure handling
11	- introducing a new dependency chain
12	- reprocessing historical data
13	- modifying publish windows or cutover behavior
14
15	## Workflow
16
17	1. Define execution semantics.
18	Specify:
19	- schedule or trigger type
20	- watermark behavior
21	- late-arriving data policy
22	- idempotency guarantees
23	- retry rules
24	- failure notification path
25
26	2. Separate normal runs from backfills.
27	Historical reprocessing should not silently behave like daily incremental runs unless that has been proven safe.
28	For `/backfill` or publish-bound replay, load `safe-backfill-and-replay-orchestration` first and complete `templates/backfill-plan.yaml` before execution.
29
30	3. Design the recovery path before rollout.
31	Include:
32	- restart behavior
33	- partial failure handling
34	- duplicate prevention
35	- publish gating
36	- rollback or pause steps
37
38	4. Estimate cost and blast radius.
39	Backfills can overload warehouses, queues, clusters, or downstream consumers.
40
41	5. Prove the run strategy.
42	Use a dry run, limited slice, or non-production environment when possible.
43
44	## Common Rationalizations
45
46	\| Rationalization \| Reality \|
47	\| --- \| --- \|
48	\| "The scheduler will handle retries for us." \| Default retries may duplicate writes or hide real data issues. \|
49	\| "Backfill is just rerunning the job for older dates." \| Historical loads often need different concurrency, checks, and cutover rules. \|
50	\| "We can figure out rollback during the incident." \| Recovery plans created during outage pressure are usually incomplete. \|
51
52	## Red Flags
53
54	- no idempotency strategy is documented
55	- backfill and incremental logic are conflated
56	- downstream consumers are not considered during replay
57	- recovery steps are absent from the plan
58
59	## Verification
60
61	- [ ] Execution, retry, and replay rules are documented
62	- [ ] Backfill behavior is explicitly designed and bounded
63	- [ ] A recovery path exists for partial or failed runs
64	- [ ] Rollout risk and cost impact have been considered

People who liked this also viewed

6 picks

Type

orchestration-and-backfills

19stars4forks

View on GitHub

How to install orchestration-and-backfills?

vaquarkhan/data-engineering-agent-skills/orchestration-and-backfills

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill orchestration-and-backfills

Installs into the current project.

Summary

Designs scheduling, reruns, and backfills safely for data systems. Use when changing orchestration, retries, dependency timing, historical reprocessing, or publish sequencing.

Files · 1

View on GitHub

SKILL.md

1	# Orchestration And Backfills
2
3	## Overview
4
5	Reliable data systems are shaped as much by rerun behavior as by transformation logic. This skill ensures schedules, retries, and backfills are deliberate, safe, and reversible.
6
7	## When to Use
8
9	- adding or changing a scheduled pipeline
10	- designing retries or failure handling
11	- introducing a new dependency chain
12	- reprocessing historical data
13	- modifying publish windows or cutover behavior
14
15	## Workflow
16
17	1. Define execution semantics.
18	Specify:
19	- schedule or trigger type
20	- watermark behavior
21	- late-arriving data policy
22	- idempotency guarantees
23	- retry rules
24	- failure notification path
25
26	2. Separate normal runs from backfills.
27	Historical reprocessing should not silently behave like daily incremental runs unless that has been proven safe.
28	For `/backfill` or publish-bound replay, load `safe-backfill-and-replay-orchestration` first and complete `templates/backfill-plan.yaml` before execution.
29
30	3. Design the recovery path before rollout.
31	Include:
32	- restart behavior
33	- partial failure handling
34	- duplicate prevention
35	- publish gating
36	- rollback or pause steps
37
38	4. Estimate cost and blast radius.
39	Backfills can overload warehouses, queues, clusters, or downstream consumers.
40
41	5. Prove the run strategy.
42	Use a dry run, limited slice, or non-production environment when possible.
43
44	## Common Rationalizations
45
46	\| Rationalization \| Reality \|
47	\| --- \| --- \|
48	\| "The scheduler will handle retries for us." \| Default retries may duplicate writes or hide real data issues. \|
49	\| "Backfill is just rerunning the job for older dates." \| Historical loads often need different concurrency, checks, and cutover rules. \|
50	\| "We can figure out rollback during the incident." \| Recovery plans created during outage pressure are usually incomplete. \|
51
52	## Red Flags
53
54	- no idempotency strategy is documented
55	- backfill and incremental logic are conflated
56	- downstream consumers are not considered during replay
57	- recovery steps are absent from the plan
58
59	## Verification
60
61	- [ ] Execution, retry, and replay rules are documented
62	- [ ] Backfill behavior is explicitly designed and bounded
63	- [ ] A recovery path exists for partial or failed runs
64	- [ ] Rollout risk and cost impact have been considered

People who liked this also viewed

6 picks

Type