weak-agent-test

Name: weak-agent-test
Author: kklimuk

145stars7forks

How to install weak-agent-test?

kklimuk/docx-cli/weak-agent-test

$npx -y skills add kklimuk/docx-cli --skill weak-agent-test

Installs into the current project.

Summary

Run the weak-agent adversarial test harness against docx-cli. Spawns weak exercise agents (Haiku by default, Sonnet to probe, or a local agent harness's pre-produced runs) to perform real document tasks over six scenarios — five editing (MNDA form-fill + font fidelity, invoice ta

Files · 1

View on GitHub

SKILL.md

1	# Adversarial review — weak-agent harness for docx-cli
2
3	This harness answers one question: **can weak agents actually use docx-cli to get
4	real work done, and what should we fix first?** It runs the `weak-agent-test`
5	workflow (`.claude/workflows/weak-agent-test.js`), which fans out one weak exercise
6	agent per scenario (Haiku by default — swappable to Sonnet via `args.model`), renders
7	every output with Microsoft Word, grades each against ground-truth criteria with an
8	opus judge, and has opus synthesize a prioritized improvement report.
9	Exercise agents do NOT self-report tool counts — every tool-economy and token number
10	is measured after the run (agents under-count their own calls ~2×, so self-reports
11	were dropped): from the agent transcripts for the Claude arms, from each scenario's
12	`exercise.json` ledger for the local arm. Both roll up into the same Run-metrics table
13	(tokens, wall-clock, tool split, correctness) via `exercise-metrics.ts`.
14
15	The test corpus is bundled with this skill under `scenarios/`, one folder per
16	scenario, named after its key (`scenarios/mnda/`, `scenarios/invoice/`, …). Each
17	scenario folder is self-describing and holds everything that scenario needs:
18
19	- `task.md` — the AGENT-FACING request, written as a human delegating the work:
20	the goal, the data, the intent — and no tool vocabulary (no `docx` commands,
21	locators, or OOXML terms), because discovering which features deliver the outcome
22	is part of what's measured,
23	- `criteria.md` — the JUDGE-ONLY grading rubric (the precise, tool-specific checks).
24	The stage step withholds it from the agent's run workspace, and the judge reads
25	it from the pristine source — the agent never sees the answer key,
26	- the fixture `.docx` to work on (edit scenarios only; authoring scenarios create
27	their output fresh),
28	- `assets/` — any additional inputs (data files, images; empty for most edit
29	scenarios).
30
31	The workflow's `SCENARIOS` manifest holds only the per-scenario routing metadata
32	(key, bucket label, `edit`/`author` kind, the doc filename); whether a baseline gets
33	rendered is DERIVED from the kind (every `edit` scenario has a pristine source, so it
34	gets one — see `hasBaseline()`), not a stored field. The actual
35	request/criteria/fixture/assets all live in the folder. The skill is
36	therefore self-contained and travels with its test corpus. To change what a scenario
37	tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx,
38	renders, reviews, the report — are dumped to `./tmp/docx-weak-agent-test/<ts>/`,
39	never into the repo.)
40
41	Staging is ONE code path for every backend: `scripts/stage-scenario.ts` copies a
42	scenario folder, strips the judge-only `criteria.md`, and verifies the inputs landed.
43	The workflow's Stage agent runs it per scenario; the local corpus runner imports it.
44
45	Each run produces, under the timestamped run dir, one result folder per scenario
46	(named after its key) plus the run-level report and metrics:
47
48	```
49	<RUN_DIR>/
50	REPORT.md ← synthesized report; the Metrics phase appends the measured
51	run-metrics section (local: in-run; Claude: your post-run pass)
52	exercise-metrics.md ← measured per-exercise-agent tokens/time/tool split
53	exercise-metrics.json
54	<key>/ ← one per scenario; the worked-on copy lives here
55	task.md assets/ ← (criteria.md is withheld from this copy — judge-only)
56	<doc>.docx ← the edited/authored document
57	renders/output/ ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
58	renders/baseline/ ← the pristine "before": page PNGs + read.md (every EDIT scenario;
59	absent only for the authored eliot-journal — no source to diff)
60	review.md ← the judge's saved review for this task (written in-run)
61	verdict.json ← the judge's structured verdict incl. t

People who liked this also viewed

6 picks

Type

weak-agent-test

145stars7forks

View on GitHub

How to install weak-agent-test?

kklimuk/docx-cli/weak-agent-test

$npx -y skills add kklimuk/docx-cli --skill weak-agent-test

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Adversarial review — weak-agent harness for docx-cli
2
3	This harness answers one question: **can weak agents actually use docx-cli to get
4	real work done, and what should we fix first?** It runs the `weak-agent-test`
5	workflow (`.claude/workflows/weak-agent-test.js`), which fans out one weak exercise
6	agent per scenario (Haiku by default — swappable to Sonnet via `args.model`), renders
7	every output with Microsoft Word, grades each against ground-truth criteria with an
8	opus judge, and has opus synthesize a prioritized improvement report.
9	Exercise agents do NOT self-report tool counts — every tool-economy and token number
10	is measured after the run (agents under-count their own calls ~2×, so self-reports
11	were dropped): from the agent transcripts for the Claude arms, from each scenario's
12	`exercise.json` ledger for the local arm. Both roll up into the same Run-metrics table
13	(tokens, wall-clock, tool split, correctness) via `exercise-metrics.ts`.
14
15	The test corpus is bundled with this skill under `scenarios/`, one folder per
16	scenario, named after its key (`scenarios/mnda/`, `scenarios/invoice/`, …). Each
17	scenario folder is self-describing and holds everything that scenario needs:
18
19	- `task.md` — the AGENT-FACING request, written as a human delegating the work:
20	the goal, the data, the intent — and no tool vocabulary (no `docx` commands,
21	locators, or OOXML terms), because discovering which features deliver the outcome
22	is part of what's measured,
23	- `criteria.md` — the JUDGE-ONLY grading rubric (the precise, tool-specific checks).
24	The stage step withholds it from the agent's run workspace, and the judge reads
25	it from the pristine source — the agent never sees the answer key,
26	- the fixture `.docx` to work on (edit scenarios only; authoring scenarios create
27	their output fresh),
28	- `assets/` — any additional inputs (data files, images; empty for most edit
29	scenarios).
30
31	The workflow's `SCENARIOS` manifest holds only the per-scenario routing metadata
32	(key, bucket label, `edit`/`author` kind, the doc filename); whether a baseline gets
33	rendered is DERIVED from the kind (every `edit` scenario has a pristine source, so it
34	gets one — see `hasBaseline()`), not a stored field. The actual
35	request/criteria/fixture/assets all live in the folder. The skill is
36	therefore self-contained and travels with its test corpus. To change what a scenario
37	tests, edit the files in its folder. (Heavy, ephemeral run outputs — edited docx,
38	renders, reviews, the report — are dumped to `./tmp/docx-weak-agent-test/<ts>/`,
39	never into the repo.)
40
41	Staging is ONE code path for every backend: `scripts/stage-scenario.ts` copies a
42	scenario folder, strips the judge-only `criteria.md`, and verifies the inputs landed.
43	The workflow's Stage agent runs it per scenario; the local corpus runner imports it.
44
45	Each run produces, under the timestamped run dir, one result folder per scenario
46	(named after its key) plus the run-level report and metrics:
47
48	```
49	<RUN_DIR>/
50	REPORT.md ← synthesized report; the Metrics phase appends the measured
51	run-metrics section (local: in-run; Claude: your post-run pass)
52	exercise-metrics.md ← measured per-exercise-agent tokens/time/tool split
53	exercise-metrics.json
54	<key>/ ← one per scenario; the worked-on copy lives here
55	task.md assets/ ← (criteria.md is withheld from this copy — judge-only)
56	<doc>.docx ← the edited/authored document
57	renders/output/ ← the OUTPUT: Word-rendered page PNGs + read.md (markdown read view)
58	renders/baseline/ ← the pristine "before": page PNGs + read.md (every EDIT scenario;
59	absent only for the authored eliot-journal — no source to diff)
60	review.md ← the judge's saved review for this task (written in-run)
61	verdict.json ← the judge's structured verdict incl. t

People who liked this also viewed

6 picks

Type