ln-34-benchmark-comparator

Name: ln-34-benchmark-comparator
Author: levnikolaevich

520stars73forks

How to install ln-34-benchmark-comparator?

levnikolaevich/claude-code-skills/ln-34-benchmark-comparator

$npx -y skills add levnikolaevich/claude-code-skills --skill ln-34-benchmark-comparator

Installs into the current project.

Summary

Compares tools or implementations through reproducible A/B workloads, correctness oracles, and controlled measurements. Use to choose alternatives; not to optimize a known bottleneck.

Files · 1

View on GitHub

SKILL.md

1	# Benchmark Comparator
2
3	Goal: Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates, setup cost, and interpretation.
4
5	Execution contract: Treat the ordered checkbox workflow below as this skill's Definition of Done. Work through every item in order, and mark it complete only when its action and required evidence are complete. `N/A`, skipped, unavailable, or delegated items remain incomplete.
6	Before returning, apply this skill's verdict, decision, and approval rules to every incomplete item and prepend Checklist: X/Y complete<br>Incomplete: None \| section/item — reason; outcome impact; exact next action; list every incomplete item.
7
8	## Tool Routing
9
10	\| Need \| Preferred tool \| Use it when \| Fallback \|
11	\|---\|---\|---\|---\|
12	\| Canonical workload and oracle \| Repository fixtures, tests, expected diffs, schemas, or independently specified outcomes \| Defining what success means before either candidate runs \| Create the smallest deterministic fixture that represents the decision \|
13	\| Isolation \| Clean Git worktrees, temporary directories, controlled environment, fixed seeds, and resettable caches \| Preventing one candidate or run from contaminating another \| Sequential clean-room setup with verified cleanup \|
14	\| Execution \| The same shell runner and wrapper for every candidate \| Capturing commands, exit status, stdout, stderr, timing, and artifacts consistently \| Manual execution with an explicit reproducibility limitation \|
15	\| Activation proof \| Logs, traces, command records, process metadata, or candidate-specific artifacts \| Verifying the intended alternative actually ran and did not fall back \| Treat the run as invalid when activation cannot be proven \|
16	\| Correctness grading \| Tests, output parser, diff, schema validation, or independent oracle \| Every scenario before cost comparison \| Manual blind grading against written expectations \|
17	\| Performance and cost \| Monotonic timer, resource metrics, token or usage telemetry, tool-call logs, and failure counts \| Metrics are observable through the same method for all candidates \| Label derived or estimated values and keep them out of measured aggregates \|
18	\| External semantics \| Official documentation and specifications \| Candidate configuration or claimed behavior needs current verification \| Primary-source web research; otherwise mark the claim `UNVERIFIED` \|
19
20	Do not tune the scenario after observing a preferred candidate, mix measurements from different workloads, or present internal estimates as externally measured facts. Benchmarking may create temporary worktrees and artifacts but must not change the source baseline or unapproved external state.
21
22	## Evidence Rules
23
24	- Specify scenarios, oracle, metrics, exclusions, and decision rule before running candidates.
25	- Hold all non-tested variables constant or record and analyze the confounder.
26	- Correctness failure cannot be compensated by better speed, token use, or cost unless the decision explicitly allows degraded correctness.
27	- Use repeated runs and report raw values, center, spread, failures, and outliers; never headline the best run.
28	- Keep setup or indexing cost, steady-state cost, maintenance burden, and runtime cost separate.
29	- Report measured, derived, estimated, and qualitative evidence in distinct fields.
30
31	## Checklist
32
33	### 1. Define the Decision and Experiment
34
35	- [ ] State the decision the benchmark must support, the candidates, intended users, representative workloads, and explicit non-goals.
36	- [ ] Include ordinary cases where a simpler or built-in candidate could reasonably win as well as cases exercising each candidate's claimed advantage; do not construct a feature demo for one side.
37	- [ ] Define scenario inputs, expected outcomes, correctness criteria, failure conditions, and an oracle independent of candidate self-report.
38	- [ ] Define primary and secondary metrics, units, measurement point, acceptance threshold, allowed tradeoffs, and tie or inconclusive rules.
39	- [ ] Identify variables that must remain fixed: repository revision, model, prompts, permissions, runtime, hardware, data, cache policy, and network conditions.
40	- [ ] Predeclare a pilot-derived repetition count tied to the minimum meaningful effect and observed noise, or a bounded sequential stopping rule with fixed error tolerance and maximum runs.
41	- [ ] Freeze and hash the scenario text, fixtures, expectations, runner, parser, and decision rule before the first candidate result is inspected.
42	- [ ] Read repository instructions and inspect Git state before creating worktrees, temporary data, or runners.
43	- [ ] Start a run-owned resource ledger with every created absolute path, worktree, process ID, cache,

levnikolaevichPublisher · 19 listings

Preview

levnikolaevich/claude-code-skills

# Benchmark Comparator

**Goal:** Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates,

**Execution contract:** Treat the ordered checkbox workflow below as this skill's Definition of Done. Work through every item in order, and mark it complete onl

Before returning, apply this skill's verdict, decision, and approval rules to every incomplete item and prepend **Checklist: X/Y complete**<br>**Incomplete: Non

Repolevnikolaevich/claude-code-skills

LicenseMIT

TypeSkills

CategoryTesting

UpdatedJul 2026

Repo stars520

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type

ln-34-benchmark-comparator

520stars73forks

View on GitHub

How to install ln-34-benchmark-comparator?

levnikolaevich/claude-code-skills/ln-34-benchmark-comparator

$npx -y skills add levnikolaevich/claude-code-skills --skill ln-34-benchmark-comparator

Installs into the current project.

Summary

Compares tools or implementations through reproducible A/B workloads, correctness oracles, and controlled measurements. Use to choose alternatives; not to optimize a known bottleneck.

Files · 1

View on GitHub

SKILL.md

1	# Benchmark Comparator
2
3	Goal: Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates, setup cost, and interpretation.
4
5	Execution contract: Treat the ordered checkbox workflow below as this skill's Definition of Done. Work through every item in order, and mark it complete only when its action and required evidence are complete. `N/A`, skipped, unavailable, or delegated items remain incomplete.
6	Before returning, apply this skill's verdict, decision, and approval rules to every incomplete item and prepend Checklist: X/Y complete<br>Incomplete: None \| section/item — reason; outcome impact; exact next action; list every incomplete item.
7
8	## Tool Routing
9
10	\| Need \| Preferred tool \| Use it when \| Fallback \|
11	\|---\|---\|---\|---\|
12	\| Canonical workload and oracle \| Repository fixtures, tests, expected diffs, schemas, or independently specified outcomes \| Defining what success means before either candidate runs \| Create the smallest deterministic fixture that represents the decision \|
13	\| Isolation \| Clean Git worktrees, temporary directories, controlled environment, fixed seeds, and resettable caches \| Preventing one candidate or run from contaminating another \| Sequential clean-room setup with verified cleanup \|
14	\| Execution \| The same shell runner and wrapper for every candidate \| Capturing commands, exit status, stdout, stderr, timing, and artifacts consistently \| Manual execution with an explicit reproducibility limitation \|
15	\| Activation proof \| Logs, traces, command records, process metadata, or candidate-specific artifacts \| Verifying the intended alternative actually ran and did not fall back \| Treat the run as invalid when activation cannot be proven \|
16	\| Correctness grading \| Tests, output parser, diff, schema validation, or independent oracle \| Every scenario before cost comparison \| Manual blind grading against written expectations \|
17	\| Performance and cost \| Monotonic timer, resource metrics, token or usage telemetry, tool-call logs, and failure counts \| Metrics are observable through the same method for all candidates \| Label derived or estimated values and keep them out of measured aggregates \|
18	\| External semantics \| Official documentation and specifications \| Candidate configuration or claimed behavior needs current verification \| Primary-source web research; otherwise mark the claim `UNVERIFIED` \|
19
20	Do not tune the scenario after observing a preferred candidate, mix measurements from different workloads, or present internal estimates as externally measured facts. Benchmarking may create temporary worktrees and artifacts but must not change the source baseline or unapproved external state.
21
22	## Evidence Rules
23
24	- Specify scenarios, oracle, metrics, exclusions, and decision rule before running candidates.
25	- Hold all non-tested variables constant or record and analyze the confounder.
26	- Correctness failure cannot be compensated by better speed, token use, or cost unless the decision explicitly allows degraded correctness.
27	- Use repeated runs and report raw values, center, spread, failures, and outliers; never headline the best run.
28	- Keep setup or indexing cost, steady-state cost, maintenance burden, and runtime cost separate.
29	- Report measured, derived, estimated, and qualitative evidence in distinct fields.
30
31	## Checklist
32
33	### 1. Define the Decision and Experiment
34
35	- [ ] State the decision the benchmark must support, the candidates, intended users, representative workloads, and explicit non-goals.
36	- [ ] Include ordinary cases where a simpler or built-in candidate could reasonably win as well as cases exercising each candidate's claimed advantage; do not construct a feature demo for one side.
37	- [ ] Define scenario inputs, expected outcomes, correctness criteria, failure conditions, and an oracle independent of candidate self-report.
38	- [ ] Define primary and secondary metrics, units, measurement point, acceptance threshold, allowed tradeoffs, and tie or inconclusive rules.
39	- [ ] Identify variables that must remain fixed: repository revision, model, prompts, permissions, runtime, hardware, data, cache policy, and network conditions.
40	- [ ] Predeclare a pilot-derived repetition count tied to the minimum meaningful effect and observed noise, or a bounded sequential stopping rule with fixed error tolerance and maximum runs.
41	- [ ] Freeze and hash the scenario text, fixtures, expectations, runner, parser, and decision rule before the first candidate result is inspected.
42	- [ ] Read repository instructions and inspect Git state before creating worktrees, temporary data, or runners.
43	- [ ] Start a run-owned resource ledger with every created absolute path, worktree, process ID, cache,

levnikolaevichPublisher · 19 listings

Preview

levnikolaevich/claude-code-skills

# Benchmark Comparator

**Goal:** Compare alternatives under controlled, reproducible conditions. Correctness comes before speed, and measured data must remain separate from estimates,

**Execution contract:** Treat the ordered checkbox workflow below as this skill's Definition of Done. Work through every item in order, and mark it complete onl

Before returning, apply this skill's verdict, decision, and approval rules to every incomplete item and prepend **Checklist: X/Y complete**<br>**Incomplete: Non

Repolevnikolaevich/claude-code-skills

LicenseMIT

TypeSkills

CategoryTesting

UpdatedJul 2026

Repo stars520

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type