incident-triage-and-pipeline-recovery

Name: incident-triage-and-pipeline-recovery
Author: vaquarkhan

19stars4forks

How to install incident-triage-and-pipeline-recovery?

vaquarkhan/data-engineering-agent-skills/incident-triage-and-pipeline-recovery

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill incident-triage-and-pipeline-recovery

Installs into the current project.

Summary

Guides agents through production data incidents. Use when a pipeline fails, publishes bad data, misses an SLA, partially loads, corrupts state, or requires rollback, replay, or stakeholder communication.

Files · 1

View on GitHub

SKILL.md

1	# Incident Triage And Pipeline Recovery
2
3	## Overview
4
5	Use this skill when production behavior is already wrong and the team needs controlled recovery. It helps agents contain blast radius, diagnose quickly, restore trust, and avoid making the incident worse.
6
7	## When to Use
8
9	- failed production runs
10	- bad or partial publishes
11	- corrupted incremental state
12	- freshness SLA breaches
13	- emergency rollback or replay decisions
14
15	Do not jump to fixes before stabilizing the system and understanding impact.
16
17	## Workflow
18
19	1. Contain the incident.
20	Decide whether to:
21	- pause schedules
22	- block downstream publish
23	- isolate bad partitions
24	- notify owners and consumers
25
26	2. Pull live signals before mutation when possible.
27	Load `mcp-data-observability-integration` to inspect lag, run state, or Spark stage metrics before reruns or replays.
28
29	3. Classify impact.
30	Identify:
31	- affected datasets
32	- time window
33	- downstream consumers
34	- whether data is late, missing, duplicated, or wrong
35
36	4. Determine the safest recovery path.
37	Options include:
38	- rerun
39	- replay
40	- rollback
41	- partial correction
42	- full backfill
43
44	When replay or backfill is chosen, load `safe-backfill-and-replay-orchestration` and draft `templates/backfill-plan.yaml` before execution.
45
46	5. Validate recovery before reopening publish paths.
47
48	6. Record the incident and add a guardrail.
49	A fixed incident with no prevention work is unfinished.
50
51	7. Turn high-value incidents into repeatable resilience tests.
52	Load `skills/data-resiliency-testing-and-failure-injection/SKILL.md` or `references/data-resiliency-testing-patterns.md` when the same failure mode must be prevented from surprising the team again.
53
54	## Common Rationalizations
55
56	\| Rationalization \| Reality \|
57	\| --- \| --- \|
58	\| "Let us rerun everything now." \| Blind reruns can duplicate data or destroy evidence. \|
59	\| "We can clean up downstream later." \| Downstream trust loss is often harder to recover than the pipeline itself. \|
60	\| "The root cause can wait." \| Without a guardrail, the same incident often returns quickly. \|
61
62	## Red Flags
63
64	- no attempt to contain blast radius
65	- reruns happen before impact is understood
66	- downstream consumers are not informed
67	- recovery succeeds but no new guardrail is added
68
69	## Verification
70
71	- [ ] Impact scope and affected consumers are identified
72	- [ ] The chosen recovery path is safer than the alternatives
73	- [ ] Publish is reopened only after validation
74	- [ ] Post-incident guardrails or checks are added

People who liked this also viewed

6 picks

Type

incident-triage-and-pipeline-recovery

19stars4forks

View on GitHub

How to install incident-triage-and-pipeline-recovery?

vaquarkhan/data-engineering-agent-skills/incident-triage-and-pipeline-recovery

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill incident-triage-and-pipeline-recovery

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Incident Triage And Pipeline Recovery
2
3	## Overview
4
5	Use this skill when production behavior is already wrong and the team needs controlled recovery. It helps agents contain blast radius, diagnose quickly, restore trust, and avoid making the incident worse.
6
7	## When to Use
8
9	- failed production runs
10	- bad or partial publishes
11	- corrupted incremental state
12	- freshness SLA breaches
13	- emergency rollback or replay decisions
14
15	Do not jump to fixes before stabilizing the system and understanding impact.
16
17	## Workflow
18
19	1. Contain the incident.
20	Decide whether to:
21	- pause schedules
22	- block downstream publish
23	- isolate bad partitions
24	- notify owners and consumers
25
26	2. Pull live signals before mutation when possible.
27	Load `mcp-data-observability-integration` to inspect lag, run state, or Spark stage metrics before reruns or replays.
28
29	3. Classify impact.
30	Identify:
31	- affected datasets
32	- time window
33	- downstream consumers
34	- whether data is late, missing, duplicated, or wrong
35
36	4. Determine the safest recovery path.
37	Options include:
38	- rerun
39	- replay
40	- rollback
41	- partial correction
42	- full backfill
43
44	When replay or backfill is chosen, load `safe-backfill-and-replay-orchestration` and draft `templates/backfill-plan.yaml` before execution.
45
46	5. Validate recovery before reopening publish paths.
47
48	6. Record the incident and add a guardrail.
49	A fixed incident with no prevention work is unfinished.
50
51	7. Turn high-value incidents into repeatable resilience tests.
52	Load `skills/data-resiliency-testing-and-failure-injection/SKILL.md` or `references/data-resiliency-testing-patterns.md` when the same failure mode must be prevented from surprising the team again.
53
54	## Common Rationalizations
55
56	\| Rationalization \| Reality \|
57	\| --- \| --- \|
58	\| "Let us rerun everything now." \| Blind reruns can duplicate data or destroy evidence. \|
59	\| "We can clean up downstream later." \| Downstream trust loss is often harder to recover than the pipeline itself. \|
60	\| "The root cause can wait." \| Without a guardrail, the same incident often returns quickly. \|
61
62	## Red Flags
63
64	- no attempt to contain blast radius
65	- reruns happen before impact is understood
66	- downstream consumers are not informed
67	- recovery succeeds but no new guardrail is added
68
69	## Verification
70
71	- [ ] Impact scope and affected consumers are identified
72	- [ ] The chosen recovery path is safer than the alternatives
73	- [ ] Publish is reopened only after validation
74	- [ ] Post-incident guardrails or checks are added

People who liked this also viewed

6 picks

Type