resilience-failure

Name: resilience-failure
Author: proyecto26

46installs1forks

How to install resilience-failure?

proyecto26/system-design-skills/resilience-failure

$npx -y skills add proyecto26/system-design-skills --skill resilience-failure

Installs into the current project.

Summary

This skill should be used when the user asks about "fault tolerance", "resilience", a "circuit breaker", "graceful degradation", "retry storm" or "thundering herd on recovery", "exponential backoff with jitter", "timeout", "bulkhead", a "single point of failure" (SPOF), "failover

Files · 1

View on GitHub

SKILL.md

1	# Resilience & Failure
2
3	Design the system so that when a part breaks — and it will — the failure is
4	contained and the user still gets a useful (if degraded) answer instead of an
5	error page or a cascading outage. Getting this wrong is the difference between a
6	slow dependency and a total meltdown: the most common amplifier of an outage is
7	the system's own reaction to it (retry storms, health-check stampedes).
8
9	## When to reach for this
10	Any design with a remote dependency, a shared resource, or an SLA. Reach here to
11	find single points of failure, decide what each call does when its dependency is
12	slow or down, protect a service from being overwhelmed (rate limiting), and plan
13	how a recovered service comes back without being crushed by the backlog.
14
15	## When NOT to
16	Don't wrap a single in-process function or a best-effort batch job in circuit
17	breakers and bulkheads — that's machinery for cross-process/cross-network calls
18	(YAGNI). Don't add retries to a non-idempotent write without an idempotency key
19	first (→ `api-design`) — you'll duplicate side effects. The cheapest design that
20	meets the availability target wins; chasing an extra nine you don't need costs
21	real complexity (→ `back-of-the-envelope` for what a nine actually buys).
22
23	## Clarify first
24	- Availability target — how many nines, and is it per-request or per-feature? (→ `back-of-the-envelope`.)
25	- Blast radius — if this dependency dies, must the whole request fail, or can the feature degrade or hide?
26	- Idempotency — is the operation safe to retry? If not, what makes it safe (key, dedup)? (→ `api-design`.)
27	- Latency budget — how long may a call wait before a timeout is better than waiting? (→ `back-of-the-envelope`.)
28	- Limit dimension & policy — rate-limit per user / IP / API key / tenant? Hard (reject) or soft (queue/shape)? Burst tolerated?
29
30	## The options
31	Layered defenses; most real designs combine several.
32
33	- Timeout — bound every remote call. Use everywhere; an unbounded wait is
34	the root of most cascades.
35	- Retry with backoff + jitter — re-attempt transient failures with growing,
36	randomized delays. Use for idempotent calls against blips; never naked retries.
37	- Circuit breaker — stop calling a dependency that's failing; fail fast and
38	probe to recover. Use when a downstream is down or slow and retries would pile on.
39	- Bulkhead — isolate resources (thread pools, connection pools, queues) per
40	dependency. Use so one slow dependency can't exhaust capacity shared by others.
41	- Graceful degradation — fall back to a cached/stale value, partial result,
42	default, or hidden feature. Use when a usable-but-worse answer beats an error.
43	- Rate limiting / load shedding — cap inbound work; reject or shape excess.
44	Use to protect a service from overload, abuse, or a stampeding caller.
45	- Redundancy / failover — run N>1 of every component; promote a standby on
46	failure. Use to remove SPOFs. (Health checks/LB failover live in `load-balancing`.)
47
48	Rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window) and
49	the circuit-breaker state machine are detailed in `references/deep-dive.md`.
50
51	## Trade-offs
52
53	\| Option \| What it solves \| What it worsens \| Change it when \|
54	\|---\|---\|---\|---\|
55	\| Timeout \| Bounds blocked threads; stops one slow call hanging the caller \| Too tight → false failures; too loose → cascades \| Tune to the dependency's p99, not a guess \|
56	\| Retry + backoff + jitter \| Rides out transient blips \| Multiplies load; duplicates non-idempotent writes \| Add jitter + cap attempts + budget; require idempotency \|
57	\| Circuit breaker \| Fails fast, gives a sick dependency room to recover \| Adds state/tuning; can trip on a blip and over-shed \| Flapping → tune thresholds / half-open probe rate \|
58	\| Bulkhead \| Contains one failure to its own pool \| Lower peak utilization; more pools to size \| One noisy dependency starves others \|
59	\| Graceful degradation \| Keeps the user served when a dependency dies \| Serves stale/partial; more code paths to test \| Correctness must be exact → fail closed instead \|
60	\| Rate limiting \| Protects the service; bounds cost/abuse \| Rejects legitimate bursts; needs shared state at scale \| Limits too strict (valid drops) or too loose (overload) \|
61	\| Redundancy / failover \| Removes SPOFs; survives node/region loss \| Cost, replication lag, failover consistency risk \| Fail

People who liked this also viewed

6 picks

Type

resilience-failure

46installs1forks

View on GitHub

How to install resilience-failure?

proyecto26/system-design-skills/resilience-failure

$npx -y skills add proyecto26/system-design-skills --skill resilience-failure

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Resilience & Failure
2
3	Design the system so that when a part breaks — and it will — the failure is
4	contained and the user still gets a useful (if degraded) answer instead of an
5	error page or a cascading outage. Getting this wrong is the difference between a
6	slow dependency and a total meltdown: the most common amplifier of an outage is
7	the system's own reaction to it (retry storms, health-check stampedes).
8
9	## When to reach for this
10	Any design with a remote dependency, a shared resource, or an SLA. Reach here to
11	find single points of failure, decide what each call does when its dependency is
12	slow or down, protect a service from being overwhelmed (rate limiting), and plan
13	how a recovered service comes back without being crushed by the backlog.
14
15	## When NOT to
16	Don't wrap a single in-process function or a best-effort batch job in circuit
17	breakers and bulkheads — that's machinery for cross-process/cross-network calls
18	(YAGNI). Don't add retries to a non-idempotent write without an idempotency key
19	first (→ `api-design`) — you'll duplicate side effects. The cheapest design that
20	meets the availability target wins; chasing an extra nine you don't need costs
21	real complexity (→ `back-of-the-envelope` for what a nine actually buys).
22
23	## Clarify first
24	- Availability target — how many nines, and is it per-request or per-feature? (→ `back-of-the-envelope`.)
25	- Blast radius — if this dependency dies, must the whole request fail, or can the feature degrade or hide?
26	- Idempotency — is the operation safe to retry? If not, what makes it safe (key, dedup)? (→ `api-design`.)
27	- Latency budget — how long may a call wait before a timeout is better than waiting? (→ `back-of-the-envelope`.)
28	- Limit dimension & policy — rate-limit per user / IP / API key / tenant? Hard (reject) or soft (queue/shape)? Burst tolerated?
29
30	## The options
31	Layered defenses; most real designs combine several.
32
33	- Timeout — bound every remote call. Use everywhere; an unbounded wait is
34	the root of most cascades.
35	- Retry with backoff + jitter — re-attempt transient failures with growing,
36	randomized delays. Use for idempotent calls against blips; never naked retries.
37	- Circuit breaker — stop calling a dependency that's failing; fail fast and
38	probe to recover. Use when a downstream is down or slow and retries would pile on.
39	- Bulkhead — isolate resources (thread pools, connection pools, queues) per
40	dependency. Use so one slow dependency can't exhaust capacity shared by others.
41	- Graceful degradation — fall back to a cached/stale value, partial result,
42	default, or hidden feature. Use when a usable-but-worse answer beats an error.
43	- Rate limiting / load shedding — cap inbound work; reject or shape excess.
44	Use to protect a service from overload, abuse, or a stampeding caller.
45	- Redundancy / failover — run N>1 of every component; promote a standby on
46	failure. Use to remove SPOFs. (Health checks/LB failover live in `load-balancing`.)
47
48	Rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window) and
49	the circuit-breaker state machine are detailed in `references/deep-dive.md`.
50
51	## Trade-offs
52
53	\| Option \| What it solves \| What it worsens \| Change it when \|
54	\|---\|---\|---\|---\|
55	\| Timeout \| Bounds blocked threads; stops one slow call hanging the caller \| Too tight → false failures; too loose → cascades \| Tune to the dependency's p99, not a guess \|
56	\| Retry + backoff + jitter \| Rides out transient blips \| Multiplies load; duplicates non-idempotent writes \| Add jitter + cap attempts + budget; require idempotency \|
57	\| Circuit breaker \| Fails fast, gives a sick dependency room to recover \| Adds state/tuning; can trip on a blip and over-shed \| Flapping → tune thresholds / half-open probe rate \|
58	\| Bulkhead \| Contains one failure to its own pool \| Lower peak utilization; more pools to size \| One noisy dependency starves others \|
59	\| Graceful degradation \| Keeps the user served when a dependency dies \| Serves stale/partial; more code paths to test \| Correctness must be exact → fail closed instead \|
60	\| Rate limiting \| Protects the service; bounds cost/abuse \| Rejects legitimate bursts; needs shared state at scale \| Limits too strict (valid drops) or too loose (overload) \|
61	\| Redundancy / failover \| Removes SPOFs; survives node/region loss \| Cost, replication lag, failover consistency risk \| Fail

People who liked this also viewed

6 picks

Type