absolute-deflake

Name: absolute-deflake
Author: maddhruv

199stars32forks

How to install absolute-deflake?

maddhruv/absolute/absolute-deflake

$npx -y skills add maddhruv/absolute --skill absolute-deflake

Installs into the current project.

Summary

Flaky test fixes: detect nondeterministic tests empirically (repeat/shuffle/parallel runs), diagnose the root cause, fix it — never retry/skip/sleep — and verify across many randomized runs. Triggers on "absolute deflake", "fix flaky tests", "CI is flaky", "this test fails random

Files · 1

View on GitHub

SKILL.md

1	> Start your first response with the 🧪 emoji.
2
3	## Absolute Deflake
4
5	Find tests that pass and fail nondeterministically, diagnose the root cause of each,
6	and fix it — not by retrying or skipping, but by removing the source of nondeterminism.
7	Output is evidence (failure rate per test) → cause → fix, verified by repeated runs.
8
9	Runs the shared engine in `references/health-engine.md` — read it for the
10	DETECT → SCAN → TRIAGE → FIX → VERIFY → REPORT loop and the safety contract. This file
11	covers only what's specific to flaky tests.
12
13	---
14
15	## When to use
16
17	- "Our CI is flaky", "this test fails randomly", "fix the intermittent failures".
18	- A test passes locally but fails in CI (or vice versa), or fails ~1 in N runs.
19	- Burning down a backlog of `retry`/`skip`-marked tests that mask real flakiness.
20
21	Not for tests that fail deterministically — that's a real bug or a real regression
22	(`/absolute work` for a fix, or just fix it). `deflake` targets nondeterministic failures.
23
24	---
25
26	## What it scans
27
28	Establish flakiness empirically — a test isn't flaky because someone said so. Use
29	`preferences.health.deflakeRuns` from config as the default N for repeat-runs (else 20):
30
31	\| Ecosystem \| Repeat-run / detect \|
32	\|---\|---\|
33	\| Jest/Vitest \| run suite N× (`--run` loop), randomize order (`--shuffle` / `testSequencer`) \|
34	\| pytest \| `pytest-randomly` + `pytest --count=N` (`pytest-repeat`); `-p no:randomly` to A/B \|
35	\| Go \| `go test -count=N -shuffle=on ./...`, `-race` \|
36
37	Also mine signals: existing `retry`/`flaky`/`skip` annotations, CI history if reachable, and
38	run the suite both in isolation and in full/parallel — order- and concurrency-
39	dependent failures only show one way. Record a failure rate per suspect test.
40
41	---
42
43	## Common root causes (diagnose, don't guess)
44
45	\| Cause \| Tell \| Fix \|
46	\|---\|---\|---\|
47	\| Test-order / shared state \| passes alone, fails in suite (or vice versa) \| isolate state; reset/teardown between tests \|
48	\| Time / clock \| fails near midnight, DST, or under load \| fake timers / inject clock; no real `sleep` \|
49	\| Async race / missing await \| fails under parallelism or slow CI \| await the actual condition; no fixed timeouts \|
50	\| Randomness \| fails ~X% with no pattern \| seed the RNG; fix the seed in tests \|
51	\| Network / external I/O \| fails offline or on slow links \| mock/stub the boundary \|
52	\| Unordered collections \| fails on map/set iteration order \| sort before asserting \|
53	\| Resource leak / port reuse \| fails on repeat or parallel runs \| unique resources; clean up \|
54
55	---
56
57	## Risk ranking (TRIAGE)
58
59	\| Wave \| Class \| Default \|
60	\|---\|---\|---\|
61	\| 1 \| clear, isolated cause (seed, await, fake clock, sort) \| fix now \|
62	\| 2 \| shared-state / ordering — needs fixture refactor \| fix this pass, per test \|
63	\| 3 \| flakiness pointing at a real product race, not just the test \| gated — surface; may be a genuine bug to fix in code \|
64
65	A flaky test sometimes means the code has a race, not the test. Don't "stabilize" the test
66	into hiding a real concurrency bug — flag wave-3 cases for a real fix.
67
68	---
69
70	## Fix & verify
71
72	- Fix the cause. Then prove it: re-run the test many times (and shuffled / parallel /
73	with `-race`) — green once is not deflaked; green across N randomized runs is.
74	- Remove the `retry`/`skip`/`flaky` annotation that was masking it once the cause is fixed.
75	- Never "fix" by adding retries, raising timeouts blindly, `sleep`, or skipping the test —
76	that hides flakiness, doesn't remove it.
77	- Re-run the full suite to confirm the fix didn't destabilize neighbors.
78
79	---
80
81	## Gotchas
82
83	1. Retry/skip as a fix. Masks the flake, ships the nondeterminism. Forbidden here.
84	2. `sleep` to dodge a race. Slows the suite and still flakes under load. Await the condition.
85	3. One green run = done. Flakes are probabilistic — verify with many randomized runs.
86	4. Stabilizing a real product race. If the code races, fix the code, not just the assertion.
87	5. Ignoring order/parallel dimension. Run isolated and in-suite; the bug hides in whichever you skip.
88
89	---
90
91	## Companion commands
92
93	- `/absolute upgrade` — a flaky suite makes upgrade verification unreliable; deflake first.
94	- `/absolute debt` — flaky-test annotations are test debt; this clears them at the root.
95	- `/absolute work` — when the flake is a genuine product-code race needing real design.

People who liked this also viewed

6 picks

Type

absolute-deflake

199stars32forks

View on GitHub

How to install absolute-deflake?

maddhruv/absolute/absolute-deflake

$npx -y skills add maddhruv/absolute --skill absolute-deflake

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	> Start your first response with the 🧪 emoji.
2
3	## Absolute Deflake
4
5	Find tests that pass and fail nondeterministically, diagnose the root cause of each,
6	and fix it — not by retrying or skipping, but by removing the source of nondeterminism.
7	Output is evidence (failure rate per test) → cause → fix, verified by repeated runs.
8
9	Runs the shared engine in `references/health-engine.md` — read it for the
10	DETECT → SCAN → TRIAGE → FIX → VERIFY → REPORT loop and the safety contract. This file
11	covers only what's specific to flaky tests.
12
13	---
14
15	## When to use
16
17	- "Our CI is flaky", "this test fails randomly", "fix the intermittent failures".
18	- A test passes locally but fails in CI (or vice versa), or fails ~1 in N runs.
19	- Burning down a backlog of `retry`/`skip`-marked tests that mask real flakiness.
20
21	Not for tests that fail deterministically — that's a real bug or a real regression
22	(`/absolute work` for a fix, or just fix it). `deflake` targets nondeterministic failures.
23
24	---
25
26	## What it scans
27
28	Establish flakiness empirically — a test isn't flaky because someone said so. Use
29	`preferences.health.deflakeRuns` from config as the default N for repeat-runs (else 20):
30
31	\| Ecosystem \| Repeat-run / detect \|
32	\|---\|---\|
33	\| Jest/Vitest \| run suite N× (`--run` loop), randomize order (`--shuffle` / `testSequencer`) \|
34	\| pytest \| `pytest-randomly` + `pytest --count=N` (`pytest-repeat`); `-p no:randomly` to A/B \|
35	\| Go \| `go test -count=N -shuffle=on ./...`, `-race` \|
36
37	Also mine signals: existing `retry`/`flaky`/`skip` annotations, CI history if reachable, and
38	run the suite both in isolation and in full/parallel — order- and concurrency-
39	dependent failures only show one way. Record a failure rate per suspect test.
40
41	---
42
43	## Common root causes (diagnose, don't guess)
44
45	\| Cause \| Tell \| Fix \|
46	\|---\|---\|---\|
47	\| Test-order / shared state \| passes alone, fails in suite (or vice versa) \| isolate state; reset/teardown between tests \|
48	\| Time / clock \| fails near midnight, DST, or under load \| fake timers / inject clock; no real `sleep` \|
49	\| Async race / missing await \| fails under parallelism or slow CI \| await the actual condition; no fixed timeouts \|
50	\| Randomness \| fails ~X% with no pattern \| seed the RNG; fix the seed in tests \|
51	\| Network / external I/O \| fails offline or on slow links \| mock/stub the boundary \|
52	\| Unordered collections \| fails on map/set iteration order \| sort before asserting \|
53	\| Resource leak / port reuse \| fails on repeat or parallel runs \| unique resources; clean up \|
54
55	---
56
57	## Risk ranking (TRIAGE)
58
59	\| Wave \| Class \| Default \|
60	\|---\|---\|---\|
61	\| 1 \| clear, isolated cause (seed, await, fake clock, sort) \| fix now \|
62	\| 2 \| shared-state / ordering — needs fixture refactor \| fix this pass, per test \|
63	\| 3 \| flakiness pointing at a real product race, not just the test \| gated — surface; may be a genuine bug to fix in code \|
64
65	A flaky test sometimes means the code has a race, not the test. Don't "stabilize" the test
66	into hiding a real concurrency bug — flag wave-3 cases for a real fix.
67
68	---
69
70	## Fix & verify
71
72	- Fix the cause. Then prove it: re-run the test many times (and shuffled / parallel /
73	with `-race`) — green once is not deflaked; green across N randomized runs is.
74	- Remove the `retry`/`skip`/`flaky` annotation that was masking it once the cause is fixed.
75	- Never "fix" by adding retries, raising timeouts blindly, `sleep`, or skipping the test —
76	that hides flakiness, doesn't remove it.
77	- Re-run the full suite to confirm the fix didn't destabilize neighbors.
78
79	---
80
81	## Gotchas
82
83	1. Retry/skip as a fix. Masks the flake, ships the nondeterminism. Forbidden here.
84	2. `sleep` to dodge a race. Slows the suite and still flakes under load. Await the condition.
85	3. One green run = done. Flakes are probabilistic — verify with many randomized runs.
86	4. Stabilizing a real product race. If the code races, fix the code, not just the assertion.
87	5. Ignoring order/parallel dimension. Run isolated and in-suite; the bug hides in whichever you skip.
88
89	---
90
91	## Companion commands
92
93	- `/absolute upgrade` — a flaky suite makes upgrade verification unreliable; deflake first.
94	- `/absolute debt` — flaky-test annotations are test debt; this clears them at the root.
95	- `/absolute work` — when the flake is a genuine product-code race needing real design.

People who liked this also viewed

6 picks

Type