data-resiliency-testing-and-failure-injection

Name: data-resiliency-testing-and-failure-injection
Author: vaquarkhan

19stars4forks

How to install data-resiliency-testing-and-failure-injection?

vaquarkhan/data-engineering-agent-skills/data-resiliency-testing-and-failure-injection

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill data-resiliency-testing-and-failure-injection

Installs into the current project.

Summary

Guides agents through resiliency testing for data platforms. Use when designing or running failure drills, recovery validation, failover tests, replay-safety checks, dependency outage exercises, or fault injection for pipelines and publishes.

Files · 1

View on GitHub

SKILL.md

1	# Data Resiliency Testing And Failure Injection
2
3	## Overview
4
5	Use this skill when the goal is to prove that a data system recovers safely under failure, not only when everything goes right. It helps agents design controlled drills for retries, restarts, dependency outages, state recovery, duplicate prevention, backlog catch-up, and publish protection.
6
7	## When to Use
8
9	- hardening a production pipeline before broad rollout
10	- testing failover, restart, replay, or checkpoint recovery behavior
11	- validating that retries do not duplicate or corrupt data
12	- proving recovery objectives for orchestrators, jobs, streams, or warehouse publishes
13	- converting a past incident into a repeatable resilience drill
14
15	Do not treat resilience testing as random breakage. The point is to validate recovery behavior with explicit safety limits and evidence.
16
17	## Workflow
18
19	1. Define the failure modes that matter.
20	Prioritize:
21	- source outage or delayed upstream delivery
22	- worker or task restart
23	- orchestrator retry and timeout behavior
24	- duplicate event or duplicate file delivery
25	- checkpoint or incremental-state recovery
26	- credential, secret, or network dependency failure
27	- partial publish or downstream unavailability
28
29	2. Define the resilience objectives.
30	Include:
31	- acceptable data loss behavior
32	- recovery time objective
33	- replay or backlog catch-up expectation
34	- duplicate-prevention requirement
35	- publish block or quarantine behavior
36	- alert and escalation expectation
37
38	3. Choose the safest drill environment.
39	Prefer:
40	- staging or isolated non-production
41	- canary datasets or partitions
42	- synthetic or masked test data
43	- bounded windows and rollback-ready test scope
44
45	4. Inject one failure mode at a time.
46	Use controlled exercises such as:
47	- killing a task or worker
48	- pausing an upstream dependency
49	- delaying input arrival
50	- replaying a duplicate input
51	- forcing an expired secret or denied permission in a safe environment
52	- simulating partial output and validating publish closure
53
54	5. Validate the recovery path.
55	Check:
56	- whether the system resumes or fails safely
57	- whether alerts fire with useful context
58	- whether duplicates are prevented
59	- whether backlog catch-up stays bounded
60	- whether publish remains blocked until validation passes
61
62	6. Record guardrails and automate the highest-value drills.
63	The best resilience test is one the team can rerun after changes, not a one-time exercise that gets forgotten.
64
65	7. Load companion skills by failure mode.
66	- replay or backfill drills: `safe-backfill-and-replay-orchestration`
67	- Kafka lag, DLQ, or schema drift: `kafka-resilience-and-schema-evolution`
68	- serverless Spark checkpoint recovery: `spark-serverless-reliability-and-state-management`
69	- live diagnosis before drills: `mcp-data-observability-integration`
70	- drill patterns: `references/data-resiliency-testing-patterns.md`
71
72	## Common Rationalizations
73
74	\| Rationalization \| Reality \|
75	\| --- \| --- \|
76	\| "If the job retries, we are resilient enough." \| Retry alone does not prove replay safety, duplicate prevention, or publish protection. \|
77	\| "We can test recovery during a real incident." \| Real incidents are the worst time to discover the recovery path is unclear or unsafe. \|
78	\| "Failure injection is too risky for data systems." \| Uncontrolled failure is riskier than bounded, reviewable drills in safe environments. \|
79	\| "The scheduler health page already proves resilience." \| Scheduler status does not prove data correctness, backlog catch-up, or downstream safety. \|
80
81	## Red Flags
82
83	- no list of prioritized failure modes exists
84	- retries are enabled without idempotency proof
85	- resilience drills have no rollback or blast-radius limits
86	- checkpoint or incremental-state recovery has never been tested
87	- alerts fire but recovery ownership is unclear
88	- a past incident has no corresponding regression drill
89
90	## Verification
91
92	- [ ] High-impact failure modes are named and prioritized
93	- [ ] Recovery objectives and acceptable failure behavior are explicit
94	- [ ] The drill scope is bounded and safe to run
95	- [ ] Recovery evidence covers alerts, replay safety, duplicate prevention, and publish protection
96	- [ ] At least one incident-derived failure mode is turned into a repeatable drill

People who liked this also viewed

6 picks

Type

data-resiliency-testing-and-failure-injection

19stars4forks

View on GitHub

How to install data-resiliency-testing-and-failure-injection?

vaquarkhan/data-engineering-agent-skills/data-resiliency-testing-and-failure-injection

$npx -y skills add vaquarkhan/data-engineering-agent-skills --skill data-resiliency-testing-and-failure-injection

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Data Resiliency Testing And Failure Injection
2
3	## Overview
4
5	Use this skill when the goal is to prove that a data system recovers safely under failure, not only when everything goes right. It helps agents design controlled drills for retries, restarts, dependency outages, state recovery, duplicate prevention, backlog catch-up, and publish protection.
6
7	## When to Use
8
9	- hardening a production pipeline before broad rollout
10	- testing failover, restart, replay, or checkpoint recovery behavior
11	- validating that retries do not duplicate or corrupt data
12	- proving recovery objectives for orchestrators, jobs, streams, or warehouse publishes
13	- converting a past incident into a repeatable resilience drill
14
15	Do not treat resilience testing as random breakage. The point is to validate recovery behavior with explicit safety limits and evidence.
16
17	## Workflow
18
19	1. Define the failure modes that matter.
20	Prioritize:
21	- source outage or delayed upstream delivery
22	- worker or task restart
23	- orchestrator retry and timeout behavior
24	- duplicate event or duplicate file delivery
25	- checkpoint or incremental-state recovery
26	- credential, secret, or network dependency failure
27	- partial publish or downstream unavailability
28
29	2. Define the resilience objectives.
30	Include:
31	- acceptable data loss behavior
32	- recovery time objective
33	- replay or backlog catch-up expectation
34	- duplicate-prevention requirement
35	- publish block or quarantine behavior
36	- alert and escalation expectation
37
38	3. Choose the safest drill environment.
39	Prefer:
40	- staging or isolated non-production
41	- canary datasets or partitions
42	- synthetic or masked test data
43	- bounded windows and rollback-ready test scope
44
45	4. Inject one failure mode at a time.
46	Use controlled exercises such as:
47	- killing a task or worker
48	- pausing an upstream dependency
49	- delaying input arrival
50	- replaying a duplicate input
51	- forcing an expired secret or denied permission in a safe environment
52	- simulating partial output and validating publish closure
53
54	5. Validate the recovery path.
55	Check:
56	- whether the system resumes or fails safely
57	- whether alerts fire with useful context
58	- whether duplicates are prevented
59	- whether backlog catch-up stays bounded
60	- whether publish remains blocked until validation passes
61
62	6. Record guardrails and automate the highest-value drills.
63	The best resilience test is one the team can rerun after changes, not a one-time exercise that gets forgotten.
64
65	7. Load companion skills by failure mode.
66	- replay or backfill drills: `safe-backfill-and-replay-orchestration`
67	- Kafka lag, DLQ, or schema drift: `kafka-resilience-and-schema-evolution`
68	- serverless Spark checkpoint recovery: `spark-serverless-reliability-and-state-management`
69	- live diagnosis before drills: `mcp-data-observability-integration`
70	- drill patterns: `references/data-resiliency-testing-patterns.md`
71
72	## Common Rationalizations
73
74	\| Rationalization \| Reality \|
75	\| --- \| --- \|
76	\| "If the job retries, we are resilient enough." \| Retry alone does not prove replay safety, duplicate prevention, or publish protection. \|
77	\| "We can test recovery during a real incident." \| Real incidents are the worst time to discover the recovery path is unclear or unsafe. \|
78	\| "Failure injection is too risky for data systems." \| Uncontrolled failure is riskier than bounded, reviewable drills in safe environments. \|
79	\| "The scheduler health page already proves resilience." \| Scheduler status does not prove data correctness, backlog catch-up, or downstream safety. \|
80
81	## Red Flags
82
83	- no list of prioritized failure modes exists
84	- retries are enabled without idempotency proof
85	- resilience drills have no rollback or blast-radius limits
86	- checkpoint or incremental-state recovery has never been tested
87	- alerts fire but recovery ownership is unclear
88	- a past incident has no corresponding regression drill
89
90	## Verification
91
92	- [ ] High-impact failure modes are named and prioritized
93	- [ ] Recovery objectives and acceptable failure behavior are explicit
94	- [ ] The drill scope is bounded and safe to run
95	- [ ] Recovery evidence covers alerts, replay safety, duplicate prevention, and publish protection
96	- [ ] At least one incident-derived failure mode is turned into a repeatable drill

People who liked this also viewed

6 picks

Type