content-refinement-agent

Name: content-refinement-agent
Author: woodfishhhh

31stars1forks

How to install content-refinement-agent?

woodfishhhh/ez_math_model/content-refinement-agent

$npx -y skills add woodfishhhh/EZ_math_model --skill content-refinement-agent

Installs into the current project.

Summary

Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER

Files · 1

View on GitHub

SKILL.md

1	# Content Refinement Agent (Step 5)
2
3	Faithful implementation of the Content Refinement Agent from PaperOrchestra
4	(Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).
5
6	Cost: ~5–7 LLM calls (App. B), typically ~3 refinement iterations, each
7	consisting of one reviewer call and one revision call.
8
9	The paper highlights this step as one of the largest contributors to overall
10	quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute
11	acceptance-rate improvement (Fig. 4). Get this step right.
12
13	## Inputs
14
15	- `workspace/drafts/paper.tex` — output of Step 4
16	- `workspace/inputs/conference_guidelines.md`
17	- `workspace/inputs/experimental_log.md` — used as ground truth for the
18	hallucination check
19	- `workspace/citation_pool.json` / `workspace/refs.bib` — the allowed
20	bibliography
21
22	## Outputs
23
24	- `workspace/refinement/iter1/`, `iter2/`, `iter3/` — per-iteration snapshots
25	containing `paper.tex`, `paper.pdf`, `review.json`, `score.json`
26	- `workspace/refinement/worklog.json` — append-only history of decisions
27	- `workspace/final/paper.tex` and `workspace/final/paper.pdf` — copy of the
28	best accepted snapshot
29
30	## The refinement loop
31
32	```
33	prev_score = score(paper.tex) # baseline from initial draft
34	snapshot iter0/
35
36	for iter in 1..ITER_CAP (default 3):
37	1. simulate_review(paper.tex) → review.json
38	(uses `references/reviewer-rubric.md` rubric)
39
40	2. apply_revision(paper.tex, review.json) → new_paper.tex
41	(uses verbatim Refinement Agent prompt at `references/prompt.md`)
42
43	3. snapshot iter<N>/ with new_paper.tex, review.json
44	latexmk -pdf new_paper.tex → iter<N>/paper.pdf
45
46	4. score(new_paper.tex) → curr_score
47
48	5. decide via score_delta.py:
49	- if curr.overall > prev.overall: ACCEPT
50	- elif curr.overall == prev.overall and net_subaxis ≥0: ACCEPT
51	- else: REVERT
52
53	6. apply_worklog.py to append the decision
54
55	7. if REVERT or no actionable weaknesses or iter == ITER_CAP: HALT
56
57	paper.tex ← new_paper.tex (only on ACCEPT)
58	prev_score ← curr_score
59
60	cp <best iter>/paper.tex → workspace/final/paper.tex
61	```
62
63	The "best" snapshot at HALT is the one with the highest accepted overall
64	score. On a REVERT halt, the best is the iteration immediately before the
65	revert.
66
67	## Step-by-step
68
69	### 0. Pre-refinement integrity gate
70
71	Before snapshotting or scoring the initial draft, run the AI failure modes gate:
72
73	Load `references/ai-failure-modes.md` (which points to `skills/shared/ai_failure_modes.md`).
74	Run all 7 checks against the draft and the inputs. This gate runs once only,
75	at the start of iteration 1.
76
77	- CONFIRMED failure → write HALT entry to worklog.json, report to user, stop.
78	- SUSPECTED failure → add WARNING comment to paper.tex, log in worklog.json, continue.
79	- No failures → proceed.
80
81	### 0b. Snapshot the initial draft
82
83	```bash
84	python skills/content-refinement-agent/scripts/snapshot.py \
85	--src workspace/drafts/paper.tex \
86	--dst workspace/refinement/iter0/
87	```
88
89	This creates `iter0/paper.tex`. Then compile to `iter0/paper.pdf`:
90
91
92	```bash
93	cd workspace/refinement/iter0/ && latexmk -pdf -interaction=nonstopmode paper.tex
94	```
95
96	Score it (see Step 1 below) → `iter0/score.json`.
97
98	### 1. Simulate peer review
99
100	For each iteration N starting from 1:
101
102	Writing quality pre-check (start of every iteration): Load
103	`references/writing-quality-check.md` and run the 5-category checklist
104	(Categories A–E) against the current draft. Note violations and add them to
105	the revision agenda.
106
107	Load `references/reviewer-rubric.md` as the system prompt for the simulated
108	reviewer call. The reviewer reads `iter<N-1>/paper.pdf` (or `paper.tex` if
109	your host LLM lacks PDF input) and produces a JSON of strengths,
110	weaknesses, questions, and per-axis scores.
111
112	The rubric is structured to mimic AgentReview (Jin et al., 2024) — the
113	paper's chosen evaluator. We ship a faithful rubric in the references
114	directory; the host agent's LLM does the actual reviewing.
115
116	Devil's Advocate reviewer: One simulated reviewer must be designated the DA
117	following `references/da-reviewer.md`. The DA challenges core claims from first
118	principles (causal overclaiming, ablation coverage, baseline fairness,
119	generalization claims, novelty inflation) rather than surface polish. If the DA
120	issues a CRITICAL finding that remains unaddressed after all reviewers weigh in,
121	that finding blocks the "refinement accepted"

People who liked this also viewed

6 picks

Type

content-refinement-agent

31stars1forks

View on GitHub

How to install content-refinement-agent?

woodfishhhh/ez_math_model/content-refinement-agent

$npx -y skills add woodfishhhh/EZ_math_model --skill content-refinement-agent

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Content Refinement Agent (Step 5)
2
3	Faithful implementation of the Content Refinement Agent from PaperOrchestra
4	(Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).
5
6	Cost: ~5–7 LLM calls (App. B), typically ~3 refinement iterations, each
7	consisting of one reviewer call and one revision call.
8
9	The paper highlights this step as one of the largest contributors to overall
10	quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute
11	acceptance-rate improvement (Fig. 4). Get this step right.
12
13	## Inputs
14
15	- `workspace/drafts/paper.tex` — output of Step 4
16	- `workspace/inputs/conference_guidelines.md`
17	- `workspace/inputs/experimental_log.md` — used as ground truth for the
18	hallucination check
19	- `workspace/citation_pool.json` / `workspace/refs.bib` — the allowed
20	bibliography
21
22	## Outputs
23
24	- `workspace/refinement/iter1/`, `iter2/`, `iter3/` — per-iteration snapshots
25	containing `paper.tex`, `paper.pdf`, `review.json`, `score.json`
26	- `workspace/refinement/worklog.json` — append-only history of decisions
27	- `workspace/final/paper.tex` and `workspace/final/paper.pdf` — copy of the
28	best accepted snapshot
29
30	## The refinement loop
31
32	```
33	prev_score = score(paper.tex) # baseline from initial draft
34	snapshot iter0/
35
36	for iter in 1..ITER_CAP (default 3):
37	1. simulate_review(paper.tex) → review.json
38	(uses `references/reviewer-rubric.md` rubric)
39
40	2. apply_revision(paper.tex, review.json) → new_paper.tex
41	(uses verbatim Refinement Agent prompt at `references/prompt.md`)
42
43	3. snapshot iter<N>/ with new_paper.tex, review.json
44	latexmk -pdf new_paper.tex → iter<N>/paper.pdf
45
46	4. score(new_paper.tex) → curr_score
47
48	5. decide via score_delta.py:
49	- if curr.overall > prev.overall: ACCEPT
50	- elif curr.overall == prev.overall and net_subaxis ≥0: ACCEPT
51	- else: REVERT
52
53	6. apply_worklog.py to append the decision
54
55	7. if REVERT or no actionable weaknesses or iter == ITER_CAP: HALT
56
57	paper.tex ← new_paper.tex (only on ACCEPT)
58	prev_score ← curr_score
59
60	cp <best iter>/paper.tex → workspace/final/paper.tex
61	```
62
63	The "best" snapshot at HALT is the one with the highest accepted overall
64	score. On a REVERT halt, the best is the iteration immediately before the
65	revert.
66
67	## Step-by-step
68
69	### 0. Pre-refinement integrity gate
70
71	Before snapshotting or scoring the initial draft, run the AI failure modes gate:
72
73	Load `references/ai-failure-modes.md` (which points to `skills/shared/ai_failure_modes.md`).
74	Run all 7 checks against the draft and the inputs. This gate runs once only,
75	at the start of iteration 1.
76
77	- CONFIRMED failure → write HALT entry to worklog.json, report to user, stop.
78	- SUSPECTED failure → add WARNING comment to paper.tex, log in worklog.json, continue.
79	- No failures → proceed.
80
81	### 0b. Snapshot the initial draft
82
83	```bash
84	python skills/content-refinement-agent/scripts/snapshot.py \
85	--src workspace/drafts/paper.tex \
86	--dst workspace/refinement/iter0/
87	```
88
89	This creates `iter0/paper.tex`. Then compile to `iter0/paper.pdf`:
90
91
92	```bash
93	cd workspace/refinement/iter0/ && latexmk -pdf -interaction=nonstopmode paper.tex
94	```
95
96	Score it (see Step 1 below) → `iter0/score.json`.
97
98	### 1. Simulate peer review
99
100	For each iteration N starting from 1:
101
102	Writing quality pre-check (start of every iteration): Load
103	`references/writing-quality-check.md` and run the 5-category checklist
104	(Categories A–E) against the current draft. Note violations and add them to
105	the revision agenda.
106
107	Load `references/reviewer-rubric.md` as the system prompt for the simulated
108	reviewer call. The reviewer reads `iter<N-1>/paper.pdf` (or `paper.tex` if
109	your host LLM lacks PDF input) and produces a JSON of strengths,
110	weaknesses, questions, and per-axis scores.
111
112	The rubric is structured to mimic AgentReview (Jin et al., 2024) — the
113	paper's chosen evaluator. We ship a faithful rubric in the references
114	directory; the host agent's LLM does the actual reviewing.
115
116	Devil's Advocate reviewer: One simulated reviewer must be designated the DA
117	following `references/da-reviewer.md`. The DA challenges core claims from first
118	principles (causal overclaiming, ablation coverage, baseline fairness,
119	generalization claims, novelty inflation) rather than surface polish. If the DA
120	issues a CRITICAL finding that remains unaddressed after all reviewers weigh in,
121	that finding blocks the "refinement accepted"

People who liked this also viewed

6 picks

Type