evaluation-anchor-checker

Name: evaluation-anchor-checker
Author: WILLOSCAR

489stars39forks

How to install evaluation-anchor-checker?

willoscar/research-units-pipeline-skills/evaluation-anchor-checker

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill evaluation-anchor-checker

Installs into the current project.

Summary

Audit and rewrite evaluation/numeric claims to ensure they carry minimal protocol context (task + metric + constraint) and avoid underspecified model naming. Trigger: evaluation anchor checker, numeric claim hygiene, underspecified numbers, protocol context, 评测锚点检查, 数字断言, 指标上

Files · 1

View on GitHub

SKILL.md

1	# Evaluation Anchor Checker (make numbers reviewer-safe)
2
3	Purpose: fix a reviewer-magnet failure mode in agent surveys:
4	- strong numeric/performance statements appear
5	- but the minimal evaluation context is missing
6
7	This skill treats numeric claims as contracts:
8	- if a number stays, the same sentence must contain enough protocol context to interpret it
9	- if that context is not in evidence, the claim must be downgraded (no guessing)
10
11	## Inputs
12
13	Preferred (pre-merge, keeps anchoring intact):
14	- the affected `sections/*.md` files
15
16	Optional context (read-only; helps you avoid guessing):
17	- `outline/writer_context_packs.jsonl` (look for `evaluation_anchor_minimal`, `evaluation_protocol`, `anchor_facts`)
18	- `outline/evidence_drafts.jsonl` / `outline/anchor_sheet.jsonl`
19	- `citations/ref.bib`
20
21	## Outputs
22
23	- Updated `sections/*.md` (or `output/DRAFT.md` if you are post-merge), with safer evaluation anchoring
24	- `output/EVAL_ANCHOR_REPORT.md` (always; short report with files checked / changed / weakened sentences)
25	- Optional completion marker: `output/eval_anchors_checked.refined.ok`
26
27	## Recommended slot in the survey pipeline
28
29	Use this as the last section-level numeric hygiene sweep before merge:
30	- after `style-harmonizer`, `opener-variator`, `section-logic-polisher`, and
31	`paragraph-curator`
32	- immediately before the final `argument-selfloop` snapshot and merge
33
34	Reason:
35	- earlier section-level rewrite passes can legitimately rephrase or fuse numeric sentences
36	- if you only wait for `pipeline-auditor`, numeric-context issues are discovered too late in the merged draft
37	- section-scoped fixes are cheaper and preserve citation anchoring better than post-merge patching
38
39	## Read Order
40
41	Always read:
42	- `references/numeric_hygiene.md`
43
44	Machine-readable asset:
45	- `assets/numeric_hygiene.json`
46
47	The asset defines the keyword families and qualitative fallback templates.
48	Keep the script deterministic and let the policy live in the asset/reference pair.
49
50	## Role prompt: Reviewer-minded Editor (evaluation hygiene)
51
52	```text
53	You are a reviewer-minded editor for evaluation claims in a technical survey.
54
55	Goal:
56	- make every numeric/performance claim interpretable and reviewer-safe
57
58	Hard constraints:
59	- do not invent numbers
60	- do not add/remove/move citation keys
61	- if protocol context is missing, weaken or remove the numeric claim
62
63	Minimum context to include when keeping a number:
64	- task / setting (what kind of task)
65	- metric (what is being measured)
66	- constraint (budget/cost/tool access/horizon/seed/logging) when relevant
67
68	Avoid:
69	- ambiguous model naming that looks hallucinated (e.g., “GPT-5”) unless the cited paper uses it verbatim
70	```
71
72	## Workflow (explicit inputs)
73
74	- Use `outline/writer_context_packs.jsonl` to locate the subsection's allowed citations and any extracted `evaluation_protocol`/`anchor_facts`.
75	- Cross-check `outline/evidence_drafts.jsonl` and `outline/anchor_sheet.jsonl` for task/metric/constraint context before touching numbers.
76	- Validate every cited key against `citations/ref.bib` (do not introduce new keys).
77	- Write `output/EVAL_ANCHOR_REPORT.md` so the pipeline has an auditable completion artifact for this sweep.
78
79	## What to enforce (the “minimum protocol trio”)
80
81	When a sentence contains digits (`%`, `x`, or numbers):
82	- Keep the number only if you can attach at least 2 of the following in the same sentence without guessing:
83	- task family / benchmark name
84	- metric definition
85	- constraint (budget, tool access, cost model, retries, horizon)
86
87	If you cannot, downgrade:
88	- remove the number and rewrite as qualitative (“often”, “can”, “may”) with the same citation
89	- or move the specificity into a verification target (“evaluations need to report …”) without adding new facts
90
91	## Mini examples (paraphrase; do not copy)
92
93	Bad (underspecified):
94	- `Model X achieves 75% exact performance [@SomeBench].`
95
96	Better (minimal context):
97	- `On <task/benchmark>, Model X reaches ~75% <metric>, under <constraint/budget/tool access> [@SomeBench].`
98
99	Better (downgrade when context is missing):
100	- `Reported gains vary, but comparisons remain fragile when budgets and retry policies are not reported [@SomeBench].`
101
102	## Done checklist
103
104	- [ ] `output/EVAL_ANCHOR_REPORT.md` exists and reports a non-zero file count.
105	- [ ] No numeric claim remains without minimal protocol context.
106	- [ ] No ambiguous model naming remains unless explicitly supported by citations.
107	- [ ] Citation keys are unchanged.
108	- [ ] If you removed/downgraded numbers, the paragraph still makes a defensible, evidence-bounded point.
109
110	## Script
111
112	### Quick Start
113
114	- `uv run python .codex/skills/evaluation-anchor-checker/scripts/run.py --workspace

People who liked this also viewed

6 picks

Type

evaluation-anchor-checker

489stars39forks

View on GitHub

How to install evaluation-anchor-checker?

willoscar/research-units-pipeline-skills/evaluation-anchor-checker

$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill evaluation-anchor-checker

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Evaluation Anchor Checker (make numbers reviewer-safe)
2
3	Purpose: fix a reviewer-magnet failure mode in agent surveys:
4	- strong numeric/performance statements appear
5	- but the minimal evaluation context is missing
6
7	This skill treats numeric claims as contracts:
8	- if a number stays, the same sentence must contain enough protocol context to interpret it
9	- if that context is not in evidence, the claim must be downgraded (no guessing)
10
11	## Inputs
12
13	Preferred (pre-merge, keeps anchoring intact):
14	- the affected `sections/*.md` files
15
16	Optional context (read-only; helps you avoid guessing):
17	- `outline/writer_context_packs.jsonl` (look for `evaluation_anchor_minimal`, `evaluation_protocol`, `anchor_facts`)
18	- `outline/evidence_drafts.jsonl` / `outline/anchor_sheet.jsonl`
19	- `citations/ref.bib`
20
21	## Outputs
22
23	- Updated `sections/*.md` (or `output/DRAFT.md` if you are post-merge), with safer evaluation anchoring
24	- `output/EVAL_ANCHOR_REPORT.md` (always; short report with files checked / changed / weakened sentences)
25	- Optional completion marker: `output/eval_anchors_checked.refined.ok`
26
27	## Recommended slot in the survey pipeline
28
29	Use this as the last section-level numeric hygiene sweep before merge:
30	- after `style-harmonizer`, `opener-variator`, `section-logic-polisher`, and
31	`paragraph-curator`
32	- immediately before the final `argument-selfloop` snapshot and merge
33
34	Reason:
35	- earlier section-level rewrite passes can legitimately rephrase or fuse numeric sentences
36	- if you only wait for `pipeline-auditor`, numeric-context issues are discovered too late in the merged draft
37	- section-scoped fixes are cheaper and preserve citation anchoring better than post-merge patching
38
39	## Read Order
40
41	Always read:
42	- `references/numeric_hygiene.md`
43
44	Machine-readable asset:
45	- `assets/numeric_hygiene.json`
46
47	The asset defines the keyword families and qualitative fallback templates.
48	Keep the script deterministic and let the policy live in the asset/reference pair.
49
50	## Role prompt: Reviewer-minded Editor (evaluation hygiene)
51
52	```text
53	You are a reviewer-minded editor for evaluation claims in a technical survey.
54
55	Goal:
56	- make every numeric/performance claim interpretable and reviewer-safe
57
58	Hard constraints:
59	- do not invent numbers
60	- do not add/remove/move citation keys
61	- if protocol context is missing, weaken or remove the numeric claim
62
63	Minimum context to include when keeping a number:
64	- task / setting (what kind of task)
65	- metric (what is being measured)
66	- constraint (budget/cost/tool access/horizon/seed/logging) when relevant
67
68	Avoid:
69	- ambiguous model naming that looks hallucinated (e.g., “GPT-5”) unless the cited paper uses it verbatim
70	```
71
72	## Workflow (explicit inputs)
73
74	- Use `outline/writer_context_packs.jsonl` to locate the subsection's allowed citations and any extracted `evaluation_protocol`/`anchor_facts`.
75	- Cross-check `outline/evidence_drafts.jsonl` and `outline/anchor_sheet.jsonl` for task/metric/constraint context before touching numbers.
76	- Validate every cited key against `citations/ref.bib` (do not introduce new keys).
77	- Write `output/EVAL_ANCHOR_REPORT.md` so the pipeline has an auditable completion artifact for this sweep.
78
79	## What to enforce (the “minimum protocol trio”)
80
81	When a sentence contains digits (`%`, `x`, or numbers):
82	- Keep the number only if you can attach at least 2 of the following in the same sentence without guessing:
83	- task family / benchmark name
84	- metric definition
85	- constraint (budget, tool access, cost model, retries, horizon)
86
87	If you cannot, downgrade:
88	- remove the number and rewrite as qualitative (“often”, “can”, “may”) with the same citation
89	- or move the specificity into a verification target (“evaluations need to report …”) without adding new facts
90
91	## Mini examples (paraphrase; do not copy)
92
93	Bad (underspecified):
94	- `Model X achieves 75% exact performance [@SomeBench].`
95
96	Better (minimal context):
97	- `On <task/benchmark>, Model X reaches ~75% <metric>, under <constraint/budget/tool access> [@SomeBench].`
98
99	Better (downgrade when context is missing):
100	- `Reported gains vary, but comparisons remain fragile when budgets and retry policies are not reported [@SomeBench].`
101
102	## Done checklist
103
104	- [ ] `output/EVAL_ANCHOR_REPORT.md` exists and reports a non-zero file count.
105	- [ ] No numeric claim remains without minimal protocol context.
106	- [ ] No ambiguous model naming remains unless explicitly supported by citations.
107	- [ ] Citation keys are unchanged.
108	- [ ] If you removed/downgraded numbers, the paragraph still makes a defensible, evidence-bounded point.
109
110	## Script
111
112	### Quick Start
113
114	- `uv run python .codex/skills/evaluation-anchor-checker/scripts/run.py --workspace

People who liked this also viewed

6 picks

Type