running-benchmarks

Name: running-benchmarks
Author: lucasfcosta

62stars1forks

How to install running-benchmarks?

lucasfcosta/backpressured/running-benchmarks

$npx -y skills add lucasfcosta/backpressured --skill running-benchmarks

Installs into the current project.

Summary

Use when a backpressured loop needs to run benchmarks on a performance-sensitive project and decide whether a change is a regression, an improvement, or a wash — per-iteration sanity checks and the full pre-done run.

Files · 1

View on GitHub

SKILL.md

1	# Running Benchmarks
2
3	## Overview
4
5	You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.
6
7	Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.
8
9	## When to Use
10
11	- A `backpressured` loop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.
12
13	Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.
14
15	## Step 1 — Discover the suites and their time budgets
16
17	If the project has a `BACKPRESSURE.md`, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — `package.json` scripts, `Makefile`, a `benchmarks/`/`bench/` dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.
18
19	## Step 2 — Pick the suite for the moment
20
21	- Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
22	- Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.
23
24	Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.
25
26	## Step 3 — Always measure baseline vs patched
27
28	Never judge from the patched numbers alone. Run the unchanged baseline (git stash, or check out the parent commit / a worktree) and your change back-to-back, on the same machine under the same conditions. The baseline is the only thing that makes the "after" number mean anything.
29
30	Quiet the machine first — close background load, don't run benchmarks mid-build or while other heavy processes compete. Uncontrolled machine state is the largest real source of noise, and "same conditions" means actually the same, not just same hardware.
31
32	## Step 4 — Beat the noise with repetition
33
34	Benchmark output is noisy (often ±3–5% run to run). One run per side is not evidence.
35
36	- Run each side multiple times (≥5 when the suite is cheap).
37	- Compare the median (and the min for CPU-bound microbenchmarks — least contended, most stable), not a single sample.
38	- Establish the noise band from the spread of the repeated runs (or a known per-suite figure). The band is your threshold for everything below.
39
40	## Step 5 — The verdict: regression / improvement / wash
41
42	\| Verdict \| Condition \| Action \|
43	\|---------\|-----------\|--------\|
44	\| Wash (neutral) \| Patched median within the noise band of baseline; distributions overlap \| Passing. Move on. \|
45	\| Improvement \| Patched consistently faster, gap clears the noise band \| Passing. Note the win. \|
46	\| Regression \| Patched consistently slower, gap clears the noise band, distributions don't overlap \| FAIL. Fix it, or stop and name the blocker — never wave it through. \|
47	\| Ambiguous \| Result straddles the band (e.g. ~noise ± a little) \| Not a verdict yet. Add runs (≥10/side) and/or run the full suite to resolve before deciding. \|
48
49	If a suspicious result still won't separate from noise after more runs — a small true shift that the suite genuinely can't resolve — do not default it to a pass. Treat it as a blocker and name it (the change may need a less noisy benchmark, or a human call), exactly as you would any other unresolved check.
50
51	## Step 6 — Record the evidence
52
53	"No regression" asserted with no numbers is not a passing check. Record baseline vs patched, run count, and the median/min you compared. Evidence before assertion — the same standard the rest of the loop holds you to.
54
55	## Common rationalizations
56
57	\| Rationalization \| Reality \|
58	\|-----------------\|---------\|
59	\| "I ran it once, the number looks fine" \| One run inside a ±5% noise band tells you nothing. Baseline + repeats or it doesn't count. \|
60	\| "It's a tiny bit slower, probably noise" \| "Probably noise" is a measurement you didn't take. I

People who liked this also viewed

6 picks

Type

running-benchmarks

62stars1forks

View on GitHub

How to install running-benchmarks?

lucasfcosta/backpressured/running-benchmarks

$npx -y skills add lucasfcosta/backpressured --skill running-benchmarks

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Running Benchmarks
2
3	## Overview
4
5	You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.
6
7	Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.
8
9	## When to Use
10
11	- A `backpressured` loop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.
12
13	Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.
14
15	## Step 1 — Discover the suites and their time budgets
16
17	If the project has a `BACKPRESSURE.md`, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — `package.json` scripts, `Makefile`, a `benchmarks/`/`bench/` dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.
18
19	## Step 2 — Pick the suite for the moment
20
21	- Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
22	- Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.
23
24	Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.
25
26	## Step 3 — Always measure baseline vs patched
27
28	Never judge from the patched numbers alone. Run the unchanged baseline (git stash, or check out the parent commit / a worktree) and your change back-to-back, on the same machine under the same conditions. The baseline is the only thing that makes the "after" number mean anything.
29
30	Quiet the machine first — close background load, don't run benchmarks mid-build or while other heavy processes compete. Uncontrolled machine state is the largest real source of noise, and "same conditions" means actually the same, not just same hardware.
31
32	## Step 4 — Beat the noise with repetition
33
34	Benchmark output is noisy (often ±3–5% run to run). One run per side is not evidence.
35
36	- Run each side multiple times (≥5 when the suite is cheap).
37	- Compare the median (and the min for CPU-bound microbenchmarks — least contended, most stable), not a single sample.
38	- Establish the noise band from the spread of the repeated runs (or a known per-suite figure). The band is your threshold for everything below.
39
40	## Step 5 — The verdict: regression / improvement / wash
41
42	\| Verdict \| Condition \| Action \|
43	\|---------\|-----------\|--------\|
44	\| Wash (neutral) \| Patched median within the noise band of baseline; distributions overlap \| Passing. Move on. \|
45	\| Improvement \| Patched consistently faster, gap clears the noise band \| Passing. Note the win. \|
46	\| Regression \| Patched consistently slower, gap clears the noise band, distributions don't overlap \| FAIL. Fix it, or stop and name the blocker — never wave it through. \|
47	\| Ambiguous \| Result straddles the band (e.g. ~noise ± a little) \| Not a verdict yet. Add runs (≥10/side) and/or run the full suite to resolve before deciding. \|
48
49	If a suspicious result still won't separate from noise after more runs — a small true shift that the suite genuinely can't resolve — do not default it to a pass. Treat it as a blocker and name it (the change may need a less noisy benchmark, or a human call), exactly as you would any other unresolved check.
50
51	## Step 6 — Record the evidence
52
53	"No regression" asserted with no numbers is not a passing check. Record baseline vs patched, run count, and the median/min you compared. Evidence before assertion — the same standard the rest of the loop holds you to.
54
55	## Common rationalizations
56
57	\| Rationalization \| Reality \|
58	\|-----------------\|---------\|
59	\| "I ran it once, the number looks fine" \| One run inside a ±5% noise band tells you nothing. Baseline + repeats or it doesn't count. \|
60	\| "It's a tiny bit slower, probably noise" \| "Probably noise" is a measurement you didn't take. I

People who liked this also viewed

6 picks

Type