pipeline-auditor
starsforks
How to install pipeline-auditor?
willoscar/research-units-pipeline-skills/pipeline-auditor
$npx -y skills add WILLOSCAR/research-units-pipeline-skills --skill pipeline-auditorAudit/regression checks for the evidence-first survey pipeline: citation health, per-section coverage, placeholder leakage, and template repetition. Trigger: auditor, audit, regression test, quality report, 审计, 回归测试. Use when: output/DRAFT.md exists and you want a deter
SKILL.md
| 1 | # Pipeline Auditor (draft audit + regression) |
| 2 | |
| 3 | Purpose: a deterministic “regression test” for the writing stage. |
| 4 | |
| 5 | It answers: |
| 6 | - did we leak placeholders or planner talk? |
| 7 | - did citation scope drift? |
| 8 | - did the draft fall back to generator voice (navigation/narration templates)? |
| 9 | - is citation density/health sufficient for a survey-like draft? |
| 10 | |
| 11 | This skill is analysis-only. It does not edit content. For all survey-family profiles, style/citation-shape violations are blocking by default. |
| 12 | |
| 13 | ## Inputs |
| 14 | |
| 15 | - `output/DRAFT.md` |
| 16 | - `outline/outline.yml` |
| 17 | - Optional (recommended): |
| 18 | - `outline/evidence_bindings.jsonl` |
| 19 | - `citations/ref.bib` |
| 20 | |
| 21 | ## Outputs |
| 22 | |
| 23 | - `output/AUDIT_REPORT.md` |
| 24 | |
| 25 | ## What it checks (deterministic) |
| 26 | |
| 27 | A150++ citation targets (used by the auditor): |
| 28 | - Per-H3: >=12 unique citations (deep: >=14). |
| 29 | - Global: >=150 unique citations across the full draft (recommended target: 165; deep floor: 165). |
| 30 | |
| 31 | Course-paper targets: |
| 32 | - Per-H3: >=4 unique citations and >=1600 non-citation characters. |
| 33 | - Global: >=24 unique citations (recommended target: 32). |
| 34 | |
| 35 | - Placeholder leakage: ellipsis (`...`, `…`), TODO markers, scaffold tags. |
| 36 | - Outline alignment: section/subsection order vs `outline/outline.yml`. |
| 37 | - Survey tables: require >=1 Markdown table for `course_paper`, >=2 for `survey`/`deep` (inserted by `section-merger` from `outline/tables_appendix.md`; index tables remain internal). |
| 38 | - Paper voice anti-patterns: |
| 39 | - narration templates (`This subsection ...`, `In this subsection ...`) |
| 40 | - slide navigation (`Next, we move ...`, `We now turn to ...`) |
| 41 | - pipeline voice (`this run`, “pipeline/stage/workspace” in prose) |
| 42 | - Evidence-policy disclaimer spam: repeated “abstract-only/title-only/provisional” boilerplate inside H3 bodies. |
| 43 | - Meta survey-guidance phrasing: `survey synthesis/comparisons should ...`. |
| 44 | - Synthesis stem repetition: repeated `Taken together, ...` and similar high-signal generator stems. |
| 45 | - Numeric claim context: numbers without minimal evaluation context tokens (benchmark/dataset/metric/budget/cost). |
| 46 | - Citation health (if `citations/ref.bib` exists): undefined keys, duplicates, basic formatting red flags. |
| 47 | - Citation-shape hard gate: no adjacent citation blocks (`[@a] [@b]`) and no duplicate keys inside one block (`[@a; @a]`). Mid-sentence citation ratio is >=20% for `course_paper` and >=30% for `survey`/`deep`. |
| 48 | - Citation scope (if `outline/evidence_bindings.jsonl` exists): citations used per H3 should stay within the bound evidence set. |
| 49 | |
| 50 | ## How to use the report (routing table) |
| 51 | |
| 52 | Treat `output/AUDIT_REPORT.md` as a “what to fix next” router. |
| 53 | |
| 54 | Common FAIL families -> responsible stage/skill: |
| 55 | |
| 56 | - Placeholders / leaked scaffolds |
| 57 | - Fix: C2–C4 artifacts are not clean. Route to `subsection-briefs` / `evidence-draft` / `writer-context-pack`, then rewrite affected sections. |
| 58 | |
| 59 | - Missing overview tables (below the selected profile's table minimum) |
| 60 | - Fix: ensure `table-schema` + `appendix-table-writer` produced `outline/tables_appendix.md` (>=1 course-paper table; >=2 survey/deep tables; citation-backed, no placeholders), then rerun `section-merger`. |
| 61 | |
| 62 | - Planner talk in transitions / narrator bridges |
| 63 | - Fix: rerun `transition-weaver` (and ensure briefs include `bridge_terms` / `contrast_hook`), then re-merge. |
| 64 | |
| 65 | - Narration templates / slide navigation inside H3 |
| 66 | - Fix: rewrite the failing `sections/S*.md` via `writer-selfloop` (local, section-level) or `subsection-polisher`. |
| 67 | |
| 68 | - Evidence-policy disclaimer spam |
| 69 | - Fix: keep evidence policy once in Intro/Related Work (front matter), delete repeats in H3 (use `draft-polisher` or local section rewrites). |
| 70 | |
| 71 | - Citation scope drift (out-of-scope bibkeys) |
| 72 | - Fix: either (a) rewrite the subsection to stay in-scope, or (b) fix mapping/bindings (`section-mapper` → `evidence-binder`) and regenerate packs. |
| 73 | |
| 74 | - Global unique citations too low |
| 75 | - Fix: `citation-diversifier` → `citation-injector` (NO NEW FACTS), then `draft-polisher`. |
| 76 | |
| 77 | - Intro/Related Work too thin / too few cites |
| 78 | - Fix: rewrite the corresponding `sections/S<sec_id>.md` front-matter file via `writer-selfloop` (front-matter path) using dense positioning + method paragraph. |
| 79 | |
| 80 | ## Prevention guidance (what upstream writers should do) |
| 81 | |
| 82 | If you want the auditor to PASS *without* a heavy polish loop: |
| 83 | - Start each H3 with a content claim + thesis (avoid narration templates). |
| 84 | - Use explicit contrasts and at least one evaluation anchor paragraph. |
| 85 | - Embed citations per claim (avoid trailing cite dumps). |
| 86 | - Put evidence-policy limitations once in the front matter, not in every H3. |
| 87 | |
| 88 | ## Script |
| 89 | |
| 90 | ### Quick Start |
| 91 | |
| 92 | - `uv run python .codex/skills/pipeline-auditor/scripts/run.py - |
People who liked this also viewed
6 picksType
- tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026
- qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026
- webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026
- playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 2026
