self-eval-bias

Name: self-eval-bias
Author: Archive228

712stars129forks

How to install self-eval-bias?

archive228/loopkit/self-eval-bias

$npx -y skills add Archive228/loopkit --skill self-eval-bias

Installs into the current project.

Summary

Detect and interrupt the pattern where an agent confidently praises work it just produced instead of reviewing it critically. Same-context grading is not review — it's rationalization.

Files · 1

View on GitHub

SKILL.md

1	# Self-Eval Bias
2
3	An agent that just produced a plan, a diff, or a report cannot fairly grade it in the same context. The reasoning that justified writing it is still loaded — every doubt was already resolved in favor of shipping. Asked to review, the same context reliably returns "looks good, ship it." This is not review. It is rationalization wearing a review's uniform.
4
5	The pattern shows up hardest in planner/generator/evaluator architectures where the evaluator drifts toward leniency over long runs — the prompts it reads fill up with the generator's reasoning, and skepticism erodes. (See Prithvi's March 2026 post on the three-agent harness: https://blog.anthropic.com/three-agent-harness-march-2026.)
6
7	## When to apply
8
9	- You just wrote code, a plan, or a claim, and the next step is "confirm it's correct".
10	- A reviewer verdict comes back positive with no cited line numbers, no failing case explored, no counter-example attempted.
11	- You're about to mark a feature `passes: true`, close an issue, or hand off to the next session.
12	- The evaluator persona in a multi-agent loop has agreed with the last N generator outputs in a row.
13
14	## Procedure
15
16	1. Notice the same-context tell. If the review verdict lands in under three sentences and contains "looks correct", "this should work", or "no issues found" without a cited artifact — treat the verdict as unwritten.
17	2. Force a fresh persona. Drop the generation context. Open a new subagent, or at minimum re-prompt with only the artifact (diff, plan, output) and the acceptance criteria — no reasoning trail, no self-justification.
18	3. Demand concrete evidence, not verdicts. The reviewer must cite: the file:line it inspected, the input it ran, the observed output, and the criterion it matched against. "LGTM" without these is a null review — discard it.
19	4. Adversarially probe. Ask the reviewer for the strongest case where the artifact fails. If it can't produce one, the review didn't happen — the reviewer just agreed.
20	5. Run the artifact. For code, exercise it end-to-end (see [[broken-window-check]]). For a plan, walk the first two steps concretely. Same-context confidence collapses fast against a runtime.
21	6. Rotate the reviewer periodically. In long multi-agent loops, re-prompt the evaluator from scratch every ~5 sprints — leniency drift compounds silently.
22
23	## Anti-patterns
24
25	- Self-review in the same turn. "Let me double-check my work" followed by immediate approval. The doubt has to cost something to be real.
26	- Praise as evidence. "This is a clean, well-structured implementation" is a vibe, not a finding. Findings cite lines.
27	- Positive verdict, empty failure_scenario. If the reviewer can't describe what a failure would look like, they didn't look for one.
28	- Rubber-stamping across a run. N consecutive "approved" verdicts from the same evaluator without a single rejection is a red flag, not a track record.
29	- Fixing the criterion instead of the artifact. Reviewer notices a gap, then edits the spec to say the gap is out of scope. The gap is in the artifact. Fix that.
30
31	## When NOT to apply
32
33	- The output is trivial and cheap to redo if wrong (a one-line rename, a config toggle).
34	- A separate reviewer with a fresh context already ran and cited concrete evidence — the check has been done, don't loop on it.
35
36	## Related
37
38	- [[broken-window-check]] — the runtime-driven version of "don't trust the last claim".
39	- [[adversarial-verify]] — the structural form of "find the strongest failure case".
40	- [[shift-notes]] — where you record what the fresh-persona review actually found.

People who liked this also viewed

6 picks

Type

tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026501k181k
test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026175k259k
qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026160k181k
verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026153k259k
webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026120k163k
playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 202664k338

self-eval-bias

712stars129forks

View on GitHub

How to install self-eval-bias?

archive228/loopkit/self-eval-bias

$npx -y skills add Archive228/loopkit --skill self-eval-bias

Installs into the current project.

Summary

Detect and interrupt the pattern where an agent confidently praises work it just produced instead of reviewing it critically. Same-context grading is not review — it's rationalization.

Files · 1

View on GitHub

SKILL.md

1	# Self-Eval Bias
2
3	An agent that just produced a plan, a diff, or a report cannot fairly grade it in the same context. The reasoning that justified writing it is still loaded — every doubt was already resolved in favor of shipping. Asked to review, the same context reliably returns "looks good, ship it." This is not review. It is rationalization wearing a review's uniform.
4
5	The pattern shows up hardest in planner/generator/evaluator architectures where the evaluator drifts toward leniency over long runs — the prompts it reads fill up with the generator's reasoning, and skepticism erodes. (See Prithvi's March 2026 post on the three-agent harness: https://blog.anthropic.com/three-agent-harness-march-2026.)
6
7	## When to apply
8
9	- You just wrote code, a plan, or a claim, and the next step is "confirm it's correct".
10	- A reviewer verdict comes back positive with no cited line numbers, no failing case explored, no counter-example attempted.
11	- You're about to mark a feature `passes: true`, close an issue, or hand off to the next session.
12	- The evaluator persona in a multi-agent loop has agreed with the last N generator outputs in a row.
13
14	## Procedure
15
16	1. Notice the same-context tell. If the review verdict lands in under three sentences and contains "looks correct", "this should work", or "no issues found" without a cited artifact — treat the verdict as unwritten.
17	2. Force a fresh persona. Drop the generation context. Open a new subagent, or at minimum re-prompt with only the artifact (diff, plan, output) and the acceptance criteria — no reasoning trail, no self-justification.
18	3. Demand concrete evidence, not verdicts. The reviewer must cite: the file:line it inspected, the input it ran, the observed output, and the criterion it matched against. "LGTM" without these is a null review — discard it.
19	4. Adversarially probe. Ask the reviewer for the strongest case where the artifact fails. If it can't produce one, the review didn't happen — the reviewer just agreed.
20	5. Run the artifact. For code, exercise it end-to-end (see [[broken-window-check]]). For a plan, walk the first two steps concretely. Same-context confidence collapses fast against a runtime.
21	6. Rotate the reviewer periodically. In long multi-agent loops, re-prompt the evaluator from scratch every ~5 sprints — leniency drift compounds silently.
22
23	## Anti-patterns
24
25	- Self-review in the same turn. "Let me double-check my work" followed by immediate approval. The doubt has to cost something to be real.
26	- Praise as evidence. "This is a clean, well-structured implementation" is a vibe, not a finding. Findings cite lines.
27	- Positive verdict, empty failure_scenario. If the reviewer can't describe what a failure would look like, they didn't look for one.
28	- Rubber-stamping across a run. N consecutive "approved" verdicts from the same evaluator without a single rejection is a red flag, not a track record.
29	- Fixing the criterion instead of the artifact. Reviewer notices a gap, then edits the spec to say the gap is out of scope. The gap is in the artifact. Fix that.
30
31	## When NOT to apply
32
33	- The output is trivial and cheap to redo if wrong (a one-line rename, a config toggle).
34	- A separate reviewer with a fresh context already ran and cited concrete evidence — the check has been done, don't loop on it.
35
36	## Related
37
38	- [[broken-window-check]] — the runtime-driven version of "don't trust the last claim".
39	- [[adversarial-verify]] — the structural form of "find the strongest failure case".
40	- [[shift-notes]] — where you record what the fresh-persona review actually found.

People who liked this also viewed

6 picks

Type