safe-backfill-and-replay-orchestration

Name: safe-backfill-and-replay-orchestration
Author: vaquarkhan

19stars4forks

How to install safe-backfill-and-replay-orchestration?

vaquarkhan/data-engineering-agent-skills/safe-backfill-and-replay-orchestration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill safe-backfill-and-replay-orchestration

Installs into the current project.

Summary

Forces replay-safe rollout plans, reconciliation gates, and rollback paths before executing any data backfill or historical reprocessing. Use when running /backfill, rerunning pipelines, repairing publish windows, or any work that risks double-counting or downstream corruption.

Files · 1

View on GitHub

SKILL.md

1	# Safe Backfill And Replay Orchestration
2
3	## Overview
4
5	Backfills are among the highest-risk data engineering operations. Agents that rerun jobs without bounded windows, publish gates, or reconciliation checks routinely cause double-counting, corrupted metrics, and downstream trust loss. This skill blocks execution until a written replay plan, rollback path, and validation gates exist.
6
7	## When to Use
8
9	- running `/backfill` or any historical reprocessing request
10	- repairing a failed or partial publish window
11	- replaying events or partitions after an incident
12	- changing transformation logic that requires recomputation
13	- cutting over to new pipeline behavior with overlapping data windows
14
15	Always load this skill before `orchestration-and-backfills` execution steps. Treat plan approval as a hard gate, not a suggestion.
16
17	## Workflow
18
19	1. Stop before execution.
20	Do not trigger reruns, delete partitions, or open publish paths until the replay plan is drafted and reviewed.
21
22	2. Run the backfill guard hook when available.
23	Execute `hooks/backfill-guard.sh` or `hooks/backfill-guard.ps1` before planning destructive or wide-impact replay work.
24
25	3. Draft the replay plan from `templates/backfill-plan.yaml`.
26	Require explicit values for:
27	- affected time window and partitions
28	- source of truth and downstream targets
29	- execution mode (`replay`, `merge`, `overwrite`) and idempotency proof
30	- publish pause state and downstream notifications
31	- rollback strategy in plain language
32	- validation checks with thresholds
33	- required approvals
34
35	4. Separate incremental from backfill semantics.
36	Confirm:
37	- concurrency limits differ from daily runs when needed
38	- watermark or cursor behavior will not skip or duplicate late data
39	- metric definitions remain comparable across the replay window
40	- downstream consumers know publish is paused
41
42	5. Define reconciliation gates before reopening publish.
43	Load `data-reconciliation-and-financial-controls` when correctness must be proven.
44	Minimum gates:
45	- row count or volume parity within threshold
46	- duplicate check on business keys
47	- metric reconciliation for business-critical aggregates
48	- contract checks on schema and nullability
49
50	6. Load companion skills by scenario.
51	- cutover or dual-run: `data-migration-and-platform-cutover`
52	- incident-driven replay: `incident-triage-and-pipeline-recovery`
53	- schedule or dependency changes: `orchestration-and-backfills`
54
55	7. Execute in bounded slices.
56	Order of operations:
57	- dry run or single-partition proof when possible
58	- expand to full window only after slice reconciliation passes
59	- reopen publish only after all gates pass
60	- record evidence in `templates/release-gate-evidence.yaml` when publish-bound
61
62	## Replay Plan Checklist
63
64	Copy and track before any execution:
65
66	```text
67	Backfill gate:
68	- [ ] templates/backfill-plan.yaml drafted with owner and reason
69	- [ ] affected window and partitions bounded
70	- [ ] publish paused and downstream owners notified
71	- [ ] idempotent write strategy documented
72	- [ ] rollback strategy documented
73	- [ ] reconciliation checks defined with thresholds
74	- [ ] approvals captured
75	- [ ] dry run or slice proof completed
76	- [ ] full window reconciliation passed
77	- [ ] publish reopened only after evidence recorded
78	```
79
80	## Common Rationalizations
81
82	\| Rationalization \| Reality \|
83	\| --- \| --- \|
84	\| "It is faster to rerun now and reconcile later." \| Late reconciliation still leaves corrupted dashboards and irreversible downstream actions. \|
85	\| "The job is idempotent so a full rerun is safe." \| Idempotency at the job level does not guarantee metric or aggregate correctness across windows. \|
86	\| "We only need to backfill one day." \| Even small windows can duplicate keys, break slowly changing dimensions, or violate publish contracts. \|
87	\| "Rollback can be improvised if metrics look wrong." \| Rollback under pressure is slow, incomplete, and often worsens blast radius. \|
88
89	## Red Flags
90
91	- execution starts without a completed backfill plan template
92	- no publish pause for downstream consumers
93	- reconciliation thresholds are undefined
94	- backfill and incremental code paths are conflated
95	- no bounded slice proof before full-window replay
96	- rollback strategy is missing or untested
97
98	## Verification
99
100	- [ ] `templates/backfill-plan.yaml` is complete and approved
101	- [ ] Publish is paused and stakeholders are notified
102	- [ ] Idempotency and write strategy are explicit at target grain
103	- [ ] Reconciliation gates passed before publish reopen
104	- [ ] Recovery evidence is recorded for audit and future incidents

People who liked this also viewed

6 picks

Type

safe-backfill-and-replay-orchestration

19stars4forks

View on GitHub

How to install safe-backfill-and-replay-orchestration?

vaquarkhan/data-engineering-agent-skills/safe-backfill-and-replay-orchestration

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill safe-backfill-and-replay-orchestration

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Safe Backfill And Replay Orchestration
2
3	## Overview
4
5	Backfills are among the highest-risk data engineering operations. Agents that rerun jobs without bounded windows, publish gates, or reconciliation checks routinely cause double-counting, corrupted metrics, and downstream trust loss. This skill blocks execution until a written replay plan, rollback path, and validation gates exist.
6
7	## When to Use
8
9	- running `/backfill` or any historical reprocessing request
10	- repairing a failed or partial publish window
11	- replaying events or partitions after an incident
12	- changing transformation logic that requires recomputation
13	- cutting over to new pipeline behavior with overlapping data windows
14
15	Always load this skill before `orchestration-and-backfills` execution steps. Treat plan approval as a hard gate, not a suggestion.
16
17	## Workflow
18
19	1. Stop before execution.
20	Do not trigger reruns, delete partitions, or open publish paths until the replay plan is drafted and reviewed.
21
22	2. Run the backfill guard hook when available.
23	Execute `hooks/backfill-guard.sh` or `hooks/backfill-guard.ps1` before planning destructive or wide-impact replay work.
24
25	3. Draft the replay plan from `templates/backfill-plan.yaml`.
26	Require explicit values for:
27	- affected time window and partitions
28	- source of truth and downstream targets
29	- execution mode (`replay`, `merge`, `overwrite`) and idempotency proof
30	- publish pause state and downstream notifications
31	- rollback strategy in plain language
32	- validation checks with thresholds
33	- required approvals
34
35	4. Separate incremental from backfill semantics.
36	Confirm:
37	- concurrency limits differ from daily runs when needed
38	- watermark or cursor behavior will not skip or duplicate late data
39	- metric definitions remain comparable across the replay window
40	- downstream consumers know publish is paused
41
42	5. Define reconciliation gates before reopening publish.
43	Load `data-reconciliation-and-financial-controls` when correctness must be proven.
44	Minimum gates:
45	- row count or volume parity within threshold
46	- duplicate check on business keys
47	- metric reconciliation for business-critical aggregates
48	- contract checks on schema and nullability
49
50	6. Load companion skills by scenario.
51	- cutover or dual-run: `data-migration-and-platform-cutover`
52	- incident-driven replay: `incident-triage-and-pipeline-recovery`
53	- schedule or dependency changes: `orchestration-and-backfills`
54
55	7. Execute in bounded slices.
56	Order of operations:
57	- dry run or single-partition proof when possible
58	- expand to full window only after slice reconciliation passes
59	- reopen publish only after all gates pass
60	- record evidence in `templates/release-gate-evidence.yaml` when publish-bound
61
62	## Replay Plan Checklist
63
64	Copy and track before any execution:
65
66	```text
67	Backfill gate:
68	- [ ] templates/backfill-plan.yaml drafted with owner and reason
69	- [ ] affected window and partitions bounded
70	- [ ] publish paused and downstream owners notified
71	- [ ] idempotent write strategy documented
72	- [ ] rollback strategy documented
73	- [ ] reconciliation checks defined with thresholds
74	- [ ] approvals captured
75	- [ ] dry run or slice proof completed
76	- [ ] full window reconciliation passed
77	- [ ] publish reopened only after evidence recorded
78	```
79
80	## Common Rationalizations
81
82	\| Rationalization \| Reality \|
83	\| --- \| --- \|
84	\| "It is faster to rerun now and reconcile later." \| Late reconciliation still leaves corrupted dashboards and irreversible downstream actions. \|
85	\| "The job is idempotent so a full rerun is safe." \| Idempotency at the job level does not guarantee metric or aggregate correctness across windows. \|
86	\| "We only need to backfill one day." \| Even small windows can duplicate keys, break slowly changing dimensions, or violate publish contracts. \|
87	\| "Rollback can be improvised if metrics look wrong." \| Rollback under pressure is slow, incomplete, and often worsens blast radius. \|
88
89	## Red Flags
90
91	- execution starts without a completed backfill plan template
92	- no publish pause for downstream consumers
93	- reconciliation thresholds are undefined
94	- backfill and incremental code paths are conflated
95	- no bounded slice proof before full-window replay
96	- rollback strategy is missing or untested
97
98	## Verification
99
100	- [ ] `templates/backfill-plan.yaml` is complete and approved
101	- [ ] Publish is paused and stakeholders are notified
102	- [ ] Idempotency and write strategy are explicit at target grain
103	- [ ] Reconciliation gates passed before publish reopen
104	- [ ] Recovery evidence is recorded for audit and future incidents

People who liked this also viewed

6 picks

Type