paper-autoraters

Name: paper-autoraters
Author: Ar9av

43installs86forks

How to install paper-autoraters?

ar9av/paperorchestra/paper-autoraters

$npx -y skills add Ar9av/PaperOrchestra --skill paper-autoraters

Installs into the current project.

Summary

Run the four paper-quality autoraters from PaperOrchestra (arXiv:2604.05018, App. F.3) — Citation F1 (P0/P1 partition + Precision/Recall/F1), Literature Review Quality (6-axis 0-100 with anti-inflation rules), SxS Overall Paper Quality (side-by-side), and SxS Literature Review Qu

Files · 1

View on GitHub

SKILL.md

1	# Paper Autoraters (App. F.3)
2
3	Faithful implementation of the four LLM-as-judge autoraters used in
4	PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).
5
6	These are the metrics the paper uses to demonstrate that PaperOrchestra
7	beats single-agent and AI-Scientist-v2 baselines. Use them to:
8
9	1. Score a generated paper against a ground-truth paper.
10	2. Compare two paper-writing pipelines side-by-side.
11	3. Validate your own host-agent execution of the paper-orchestra pipeline.
12
13	## The four autoraters
14
15	\| Autorater \| What it does \| Inputs \| Output \|
16	\|---\|---\|---\|---\|
17	\| Citation F1 — P0/P1 partition \| Partitions reference list into P0 (must-cite) and P1 (good-to-cite) given the paper text \| one paper text + its references list \| JSON `{ref_num: "P0"\\|"P1"}` \|
18	\| Literature Review Quality \| 6-axis 0-100 score for Intro+Related Work, with anti-inflation hard caps \| one paper PDF/text + reference avg citation count \| JSON with `axis_scores`, `penalties`, `summary`, `overall_score` \|
19	\| SxS Overall Paper Quality \| Holistic side-by-side preference judgment \| two papers (PDF or text) \| JSON with `winner` ∈ {paper_1, paper_2, tie} \|
20	\| SxS Literature Review Quality \| Side-by-side preference, Intro+Related Work only \| two papers \| JSON with `winner` ∈ {paper_1, paper_2, tie} \|
21
22	The paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0
23	(Gemini) or default 1.0 (GPT-5, which doesn't allow temperature
24	adjustment). Use whatever your host LLM is.
25
26	## Workflow
27
28	### Citation F1 (compute Precision / Recall / F1 vs ground truth)
29
30	This is a two-step procedure:
31
32	#### Step 1: Partition the reference lists into P0 / P1
33
34	For both the ground-truth paper AND the generated paper, run the LLM with
35	`references/citation-f1-prompt.md`:
36
37	```
38	inputs:
39	paper_text: full paper LaTeX or markdown
40	references_str: numbered reference list (e.g., "1. Vaswani et al. (2017)
41	Attention Is All You Need. NeurIPS. 2. He et al. (2016)
42	Deep Residual Learning for Image Recognition. CVPR. ...")
43
44	output: JSON {"1": "P0", "2": "P1", "3": "P0", ...}
45	```
46
47	Save both partitions:
48	- `bench/<paper_id>/gt_partition.json`
49	- `bench/<paper_id>/gen_partition.json`
50
51	#### Step 2: Resolve references to entity IDs and compute F1
52
53	The paper uses Semantic Scholar paper IDs to match references between the
54	two lists. The `compute_f1.py` script does this deterministically given
55	two input lists:
56
57	```bash
58	python skills/paper-autoraters/scripts/compute_f1.py \
59	--gt-partition gt_partition.json \
60	--gt-refs gt_refs.json \
61	--gen-partition gen_partition.json \
62	--gen-refs gen_refs.json \
63	--out f1_report.json
64	```
65
66	Where `gt_refs.json` and `gen_refs.json` are lists of `{ref_num,
67	paper_id, title}` produced by your host's S2-resolution pass (the same
68	fuzzy match + S2 verification used by `literature-review-agent/scripts/`).
69
70	Output JSON contains P0 / P1 / overall Precision, Recall, F1.
71
72	### Literature Review Quality (single paper, 6 axes)
73
74	Load `references/litreview-quality-prompt.md`. Inputs:
75
76	- The full paper PDF (or LaTeX/markdown if your host lacks PDF input)
77	- `avg_citation_count` for the venue/field (used as the baseline for
78	citation count anchoring, e.g., 58.52 for CVPR 2025, 59.18 for ICLR 2025
79	per the paper)
80
81	The prompt instructs the model to evaluate ONLY the literature-review
82	function of the paper (Introduction + Related Work / Background sections).
83	It produces a strict JSON output with per-axis scores and justifications.
84
85	Critical anti-inflation rules baked into the prompt:
86
87	\| Rule \| Cap \|
88	\|---\|---\|
89	\| Default expectation \| overall 45-70 \|
90	\| > 85 requires strong evidence on ALL axes \| — \|
91	\| > 90 extremely rare (near-survey-level mastery) \| — \|
92	\| Any axis < 50 → overall rarely > 75 \| — \|
93	\| Mostly descriptive review \| Critical Analysis ≤ 60 \|
94	\| Novelty asserted without comparison \| Positioning ≤ 60 \|
95	\| Sparse/inconsistent citations \| Citation Rigor ≤ 60 \|
96	\| Citation count < 50% of avg \| Coverage ≤ 55 \|
97	\| Citation count > 120% of avg \| Coverage = "strong" \|
98
99	Plus penalty table:
100
101	\| Penalty \| Range \|
102	\|---\|---\|
103	\| Overclaiming novelty \| -5 to -15 \|
104	\| Missing key recent work \| -5 to -15 \|
105	\| Mostly descriptive review \| -5 to -10 \|
106	\| Weak gap statements \| -5 to -10 \|
107	\| Citation dumping \| -5 to -10 \|
108
109	Save the output to `litreview_quality_score.json`. The score JSON is the
110	same shape used by `content-refinement-agent/scripts/score_delta.py`, so
111	you can re-use the halt-rule logic to compare iterations.
112
113	### SxS Overall

People who liked this also viewed

6 picks

Type

paper-autoraters

43installs86forks

View on GitHub

How to install paper-autoraters?

ar9av/paperorchestra/paper-autoraters

$npx -y skills add Ar9av/PaperOrchestra --skill paper-autoraters

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Paper Autoraters (App. F.3)
2
3	Faithful implementation of the four LLM-as-judge autoraters used in
4	PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).
5
6	These are the metrics the paper uses to demonstrate that PaperOrchestra
7	beats single-agent and AI-Scientist-v2 baselines. Use them to:
8
9	1. Score a generated paper against a ground-truth paper.
10	2. Compare two paper-writing pipelines side-by-side.
11	3. Validate your own host-agent execution of the paper-orchestra pipeline.
12
13	## The four autoraters
14
15	\| Autorater \| What it does \| Inputs \| Output \|
16	\|---\|---\|---\|---\|
17	\| Citation F1 — P0/P1 partition \| Partitions reference list into P0 (must-cite) and P1 (good-to-cite) given the paper text \| one paper text + its references list \| JSON `{ref_num: "P0"\\|"P1"}` \|
18	\| Literature Review Quality \| 6-axis 0-100 score for Intro+Related Work, with anti-inflation hard caps \| one paper PDF/text + reference avg citation count \| JSON with `axis_scores`, `penalties`, `summary`, `overall_score` \|
19	\| SxS Overall Paper Quality \| Holistic side-by-side preference judgment \| two papers (PDF or text) \| JSON with `winner` ∈ {paper_1, paper_2, tie} \|
20	\| SxS Literature Review Quality \| Side-by-side preference, Intro+Related Work only \| two papers \| JSON with `winner` ∈ {paper_1, paper_2, tie} \|
21
22	The paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0
23	(Gemini) or default 1.0 (GPT-5, which doesn't allow temperature
24	adjustment). Use whatever your host LLM is.
25
26	## Workflow
27
28	### Citation F1 (compute Precision / Recall / F1 vs ground truth)
29
30	This is a two-step procedure:
31
32	#### Step 1: Partition the reference lists into P0 / P1
33
34	For both the ground-truth paper AND the generated paper, run the LLM with
35	`references/citation-f1-prompt.md`:
36
37	```
38	inputs:
39	paper_text: full paper LaTeX or markdown
40	references_str: numbered reference list (e.g., "1. Vaswani et al. (2017)
41	Attention Is All You Need. NeurIPS. 2. He et al. (2016)
42	Deep Residual Learning for Image Recognition. CVPR. ...")
43
44	output: JSON {"1": "P0", "2": "P1", "3": "P0", ...}
45	```
46
47	Save both partitions:
48	- `bench/<paper_id>/gt_partition.json`
49	- `bench/<paper_id>/gen_partition.json`
50
51	#### Step 2: Resolve references to entity IDs and compute F1
52
53	The paper uses Semantic Scholar paper IDs to match references between the
54	two lists. The `compute_f1.py` script does this deterministically given
55	two input lists:
56
57	```bash
58	python skills/paper-autoraters/scripts/compute_f1.py \
59	--gt-partition gt_partition.json \
60	--gt-refs gt_refs.json \
61	--gen-partition gen_partition.json \
62	--gen-refs gen_refs.json \
63	--out f1_report.json
64	```
65
66	Where `gt_refs.json` and `gen_refs.json` are lists of `{ref_num,
67	paper_id, title}` produced by your host's S2-resolution pass (the same
68	fuzzy match + S2 verification used by `literature-review-agent/scripts/`).
69
70	Output JSON contains P0 / P1 / overall Precision, Recall, F1.
71
72	### Literature Review Quality (single paper, 6 axes)
73
74	Load `references/litreview-quality-prompt.md`. Inputs:
75
76	- The full paper PDF (or LaTeX/markdown if your host lacks PDF input)
77	- `avg_citation_count` for the venue/field (used as the baseline for
78	citation count anchoring, e.g., 58.52 for CVPR 2025, 59.18 for ICLR 2025
79	per the paper)
80
81	The prompt instructs the model to evaluate ONLY the literature-review
82	function of the paper (Introduction + Related Work / Background sections).
83	It produces a strict JSON output with per-axis scores and justifications.
84
85	Critical anti-inflation rules baked into the prompt:
86
87	\| Rule \| Cap \|
88	\|---\|---\|
89	\| Default expectation \| overall 45-70 \|
90	\| > 85 requires strong evidence on ALL axes \| — \|
91	\| > 90 extremely rare (near-survey-level mastery) \| — \|
92	\| Any axis < 50 → overall rarely > 75 \| — \|
93	\| Mostly descriptive review \| Critical Analysis ≤ 60 \|
94	\| Novelty asserted without comparison \| Positioning ≤ 60 \|
95	\| Sparse/inconsistent citations \| Citation Rigor ≤ 60 \|
96	\| Citation count < 50% of avg \| Coverage ≤ 55 \|
97	\| Citation count > 120% of avg \| Coverage = "strong" \|
98
99	Plus penalty table:
100
101	\| Penalty \| Range \|
102	\|---\|---\|
103	\| Overclaiming novelty \| -5 to -15 \|
104	\| Missing key recent work \| -5 to -15 \|
105	\| Mostly descriptive review \| -5 to -10 \|
106	\| Weak gap statements \| -5 to -10 \|
107	\| Citation dumping \| -5 to -10 \|
108
109	Save the output to `litreview_quality_score.json`. The score JSON is the
110	same shape used by `content-refinement-agent/scripts/score_delta.py`, so
111	you can re-use the halt-rule logic to compare iterations.
112
113	### SxS Overall

People who liked this also viewed

6 picks

Type