paper-writing-bench

Name: paper-writing-bench
Author: Ar9av

41installs86forks

How to install paper-writing-bench?

ar9av/paperorchestra/paper-writing-bench

$npx -y skills add Ar9av/PaperOrchestra --skill paper-writing-bench

Installs into the current project.

Summary

Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGG

Files · 1

View on GitHub

SKILL.md

1	# PaperWritingBench (§3)
2
3	Faithful implementation of the PaperWritingBench dataset construction
4	procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and
5	App. C, F.2).
6
7	The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025).
8	For each paper, the authors reverse-engineer the (I, E) tuple by stripping
9	narrative flow from the original PDF using the three prompts in App. F.2.
10	You can use this skill to reverse-engineer your own benchmark cases from
11	any paper PDF.
12
13	## What this skill does
14
15	Given an existing AI research paper (PDF or markdown extract), produce:
16
17	- `idea.md` (Sparse variant) — high-level concept note, no math, no
18	experimental results
19	- `idea.md` (Dense variant) — detailed technical proposal with LaTeX
20	equations and variable definitions, but still no experimental results
21	- `experimental_log.md` — exhaustive raw experimental setup, numeric data,
22	and qualitative observations, with all narrative references stripped
23
24	These three files form a complete (I, E) input pair for the
25	paper-orchestra pipeline. You can then run the pipeline and compare its
26	output to the original paper using `paper-autoraters`.
27
28	## Inputs
29
30	- A paper PDF or extracted markdown text. The paper uses MinerU
31	(Wang et al., 2024) for PDF→markdown extraction; you (the host agent)
32	should use whatever PDF extractor your environment provides.
33	- For controlled experiments, you may also extract figures separately
34	(PDFFigures 2.0 in the paper).
35
36	## Outputs
37
38	- `bench/<paper_id>/idea_sparse.md` — Sparse variant
39	- `bench/<paper_id>/idea_dense.md` — Dense variant
40	- `bench/<paper_id>/experimental_log.md` — Experimental log
41
42	## Workflow
43
44	For each paper, run three independent LLM calls using the verbatim prompts
45	below:
46
47	### 1. Sparse idea generation
48
49	Load `references/sparse-idea-prompt.md`. Pass the paper text (or
50	markdown extract) as `{paper_content}`. The prompt instructs the model to:
51
52	- Stop extracting at empirical verification (no Experiments / Results / Comparisons)
53	- Use first-person future tense ("We propose to explore...")
54	- Avoid LaTeX math; describe components by function
55	- Anonymize authors and titles
56
57	Output: `idea_sparse.md` with the four sections (Problem Statement, Core
58	Hypothesis, Proposed Methodology high-level, Expected Contribution).
59
60	### 2. Dense idea generation
61
62	Load `references/dense-idea-prompt.md`. Same input. The prompt instructs
63	the model to:
64
65	- Preserve mathematical formulations using LaTeX
66	- Define every variable used in equations
67	- Include specific architectural choices and dimensions
68	- Same exclusion zone (no experiments)
69
70	Output: `idea_dense.md` with the four sections (Problem Statement, Core
71	Hypothesis, Proposed Methodology detailed, Expected Contribution).
72
73	### 3. Experimental log generation
74
75	Load `references/experimental-log-prompt.md`. Same input. The prompt
76	instructs the model to:
77
78	- Use past-tense persona ("We ran...", "The results were...")
79	- Strip all references to figure/table numbers
80	- Deconstruct tables into raw numeric data
81	- Log figure findings as factual observations
82	- Anonymize authors
83
84	Output: `experimental_log.md` with sections for Setup, Raw Numeric Data,
85	and Qualitative Observations.
86
87	## Critical rules from the prompts
88
89	These are excerpted from App. F.2. The host agent MUST honor them:
90
91	- No citations. None of the three outputs may contain `\cite`,
92	reference numbers, or author names from the source paper.
93	- No URLs. Strip all hyperlinks.
94	- Anonymize. Author identities, affiliations, acknowledgements all
95	removed.
96	- Self-contained. Each file must make sense without the original paper.
97	- No experimental leakage in idea files. The Sparse and Dense ideas
98	must stop where empirical verification begins. They describe what will
99	be done, not what was done.
100	- No table/figure references in experimental log. No "as shown in
101	Table 1", "see Fig. 5". The downstream paper-orchestra pipeline will
102	generate its own figures and tables — the log must not assume any
103	particular ones exist.
104	- 100% numeric accuracy in experimental log. This becomes the ground
105	truth for the section-writing-agent and content-refinement-agent's
106	hallucination check.
107
108	## How the bench is used
109
110	After producing `(idea_sparse.md, idea_dense.md, experimental_log.md)` for
111	a paper:
112
113	1. Pick a variant (Sparse or Dense) — the paper ablates both, with Dense
114	producing more rigorous methodology and Sparse exercising the system's
115	robustness on under-specified inputs.
116	2. Drop the chosen `ide

People who liked this also viewed

6 picks

Type

paper-writing-bench

41installs86forks

View on GitHub

How to install paper-writing-bench?

ar9av/paperorchestra/paper-writing-bench

$npx -y skills add Ar9av/PaperOrchestra --skill paper-writing-bench

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# PaperWritingBench (§3)
2
3	Faithful implementation of the PaperWritingBench dataset construction
4	procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and
5	App. C, F.2).
6
7	The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025).
8	For each paper, the authors reverse-engineer the (I, E) tuple by stripping
9	narrative flow from the original PDF using the three prompts in App. F.2.
10	You can use this skill to reverse-engineer your own benchmark cases from
11	any paper PDF.
12
13	## What this skill does
14
15	Given an existing AI research paper (PDF or markdown extract), produce:
16
17	- `idea.md` (Sparse variant) — high-level concept note, no math, no
18	experimental results
19	- `idea.md` (Dense variant) — detailed technical proposal with LaTeX
20	equations and variable definitions, but still no experimental results
21	- `experimental_log.md` — exhaustive raw experimental setup, numeric data,
22	and qualitative observations, with all narrative references stripped
23
24	These three files form a complete (I, E) input pair for the
25	paper-orchestra pipeline. You can then run the pipeline and compare its
26	output to the original paper using `paper-autoraters`.
27
28	## Inputs
29
30	- A paper PDF or extracted markdown text. The paper uses MinerU
31	(Wang et al., 2024) for PDF→markdown extraction; you (the host agent)
32	should use whatever PDF extractor your environment provides.
33	- For controlled experiments, you may also extract figures separately
34	(PDFFigures 2.0 in the paper).
35
36	## Outputs
37
38	- `bench/<paper_id>/idea_sparse.md` — Sparse variant
39	- `bench/<paper_id>/idea_dense.md` — Dense variant
40	- `bench/<paper_id>/experimental_log.md` — Experimental log
41
42	## Workflow
43
44	For each paper, run three independent LLM calls using the verbatim prompts
45	below:
46
47	### 1. Sparse idea generation
48
49	Load `references/sparse-idea-prompt.md`. Pass the paper text (or
50	markdown extract) as `{paper_content}`. The prompt instructs the model to:
51
52	- Stop extracting at empirical verification (no Experiments / Results / Comparisons)
53	- Use first-person future tense ("We propose to explore...")
54	- Avoid LaTeX math; describe components by function
55	- Anonymize authors and titles
56
57	Output: `idea_sparse.md` with the four sections (Problem Statement, Core
58	Hypothesis, Proposed Methodology high-level, Expected Contribution).
59
60	### 2. Dense idea generation
61
62	Load `references/dense-idea-prompt.md`. Same input. The prompt instructs
63	the model to:
64
65	- Preserve mathematical formulations using LaTeX
66	- Define every variable used in equations
67	- Include specific architectural choices and dimensions
68	- Same exclusion zone (no experiments)
69
70	Output: `idea_dense.md` with the four sections (Problem Statement, Core
71	Hypothesis, Proposed Methodology detailed, Expected Contribution).
72
73	### 3. Experimental log generation
74
75	Load `references/experimental-log-prompt.md`. Same input. The prompt
76	instructs the model to:
77
78	- Use past-tense persona ("We ran...", "The results were...")
79	- Strip all references to figure/table numbers
80	- Deconstruct tables into raw numeric data
81	- Log figure findings as factual observations
82	- Anonymize authors
83
84	Output: `experimental_log.md` with sections for Setup, Raw Numeric Data,
85	and Qualitative Observations.
86
87	## Critical rules from the prompts
88
89	These are excerpted from App. F.2. The host agent MUST honor them:
90
91	- No citations. None of the three outputs may contain `\cite`,
92	reference numbers, or author names from the source paper.
93	- No URLs. Strip all hyperlinks.
94	- Anonymize. Author identities, affiliations, acknowledgements all
95	removed.
96	- Self-contained. Each file must make sense without the original paper.
97	- No experimental leakage in idea files. The Sparse and Dense ideas
98	must stop where empirical verification begins. They describe what will
99	be done, not what was done.
100	- No table/figure references in experimental log. No "as shown in
101	Table 1", "see Fig. 5". The downstream paper-orchestra pipeline will
102	generate its own figures and tables — the log must not assume any
103	particular ones exist.
104	- 100% numeric accuracy in experimental log. This becomes the ground
105	truth for the section-writing-agent and content-refinement-agent's
106	hallucination check.
107
108	## How the bench is used
109
110	After producing `(idea_sparse.md, idea_dense.md, experimental_log.md)` for
111	a paper:
112
113	1. Pick a variant (Sparse or Dense) — the paper ablates both, with Dense
114	producing more rigorous methodology and Sparse exercising the system's
115	robustness on under-specified inputs.
116	2. Drop the chosen `ide

People who liked this also viewed

6 picks

Type