human-eval-handoff-repair

Name: human-eval-handoff-repair
Author: yha9806

24stars3forks

How to install human-eval-handoff-repair?

yha9806/academic-writing-toolkit/human-eval-handoff-repair

$npx -y skills add yha9806/academic-writing-toolkit --skill human-eval-handoff-repair

Installs into the current project.

Summary

Use when validating, repairing, or mapping human-evaluation handoff packages, filled annotation CSVs, rebuttal annotation UIs, or reviewer annotation returns across package versions. Applies to checking row alignment, detecting cross-snapshot contamination, converting old labels

Files · 1

View on GitHub

SKILL.md

1	# Human Eval Handoff Repair
2
3	Use this skill when the user asks to inspect, repair, migrate, or validate human-evaluation packages or filled annotation CSVs, especially when multiple package versions exist.
4
5	## First Principle
6
7	Never trust row number, filename, or visible ID alone. Treat a filled CSV as usable only after it is aligned to the target public package by stable task-specific keys and its editable labels pass schema checks.
8
9	Do not fix labels by guessing. If a row cannot be matched safely, leave its annotation fields blank and produce a refill/import template plus an unmatched reference file.
10
11	## Inputs To Locate
12
13	- Target public handoff package folder or ZIP.
14	- Filled CSVs from annotators.
15	- Any older package claimed as the source version.
16	- Current task schemas from the target UI data or target task CSVs.
17	- If relevant, QC reports from the target package.
18
19	Prefer a user-specified output directory. If none is specified, write generated reports and repaired files under `codex_outputs/` in the current workspace. Avoid synced personal document folders unless the user explicitly asks for them.
20
21	## Task Types And Stable Keys
22
23	Use these stable keys before copying any labels:
24
25	- Claim Warrant Audit: `item_id`, `tradition`, `medium`, `period`, `layer`, `dimension_id`, `claim_text`.
26	- Release Suitability Audit: `item_id`, `source_bucket`, `pipeline_mode`, `option_a_source`, `option_b_source`, `governed_option`, `option_a_text`, `option_b_text`.
27	- Card Faithfulness Spot Audit: `audit_field_id`, `tradition`, `card_type`, `dimension_id`, `field_name`, `field_value`, `source_reference_excerpt`.
28
29	If the exact stable key does not match, do not formally aggregate the row. Text-only fuzzy candidates may be generated for manual review, but must not be treated as validated mappings.
30
31	## Editable Label Columns
32
33	Claim Warrant Audit editable columns:
34
35	- `support_source`
36	- `overreach`
37	- `certainty_calibration`
38	- `recommended_action`
39	- `annotator_confidence`
40	- `notes`
41
42	Release Suitability Audit editable columns:
43
44	- `suitability_decision`
45	- `reason_primary`
46	- `risk_in_worse_option`
47	- `annotator_confidence`
48	- `notes`
49
50	Card Faithfulness Spot Audit editable columns:
51
52	- `field_status`
53	- `field_provenance`
54	- `notes`
55
56	Only copy editable label columns. Never copy old evidence text, image paths, metadata fields, or UI-only columns into a new package.
57
58	## Label Schema Checks
59
60	Claim legal values:
61
62	- `support_source`: `visual_observation`, `metadata_context`, `card_or_expert_context`, `b0_excerpt`, `mixed`, `none`
63	- `overreach`: `no_overreach`, `minor_overreach`, `major_overreach`, `unsupported`
64	- `certainty_calibration`: `well_calibrated`, `slightly_overstated`, `clearly_overstated`, `uncertain_or_not_checkable`
65	- `recommended_action`: `accept`, `revise`, `reject`, `filter_or_abstain`
66	- `annotator_confidence`: usually `1` to `5`
67
68	Release legal values:
69
70	- `suitability_decision`: `A_better`, `B_better`, `tie_both_ok`, `tie_both_bad`, `abstain_required`
71	- `reason_primary`: target package enum only; validate from the target CSV or UI schema.
72	- `risk_in_worse_option`: target package enum only; validate from the target CSV or UI schema.
73	- `annotator_confidence`: usually `1` to `5`
74
75	Card legal values:
76
77	- `field_status`: `faithful`, `over_specific`, `unsupported`, `wrong`, `unclear`
78	- `field_provenance`: `expert_critique`, `metadata`, `visual_observation`, `accepted_domain_context`, `none`, `unclear`
79
80	Normalize harmless spelling and casing only when the mapping is deterministic. Otherwise flag the value and leave it blank.
81
82	## Required QC Procedure
83
84	For every filled CSV:
85
86	1. Count rows against the target task row count.
87	2. Check required label columns are present.
88	3. Check all non-empty labels are legal enum values.
89	4. Compare non-label/source columns with the target blank CSV.
90	5. Compute exact stable-key overlap with the target package.
91	6. Identify whether the filled CSV matches the current package, an older package, or neither.
92	7. Run simple logic checks.
93	8. Produce a QC JSON and Markdown summary.
94
95	Important logic flags:
96
97	- Claim: `support_source=none` with `recommended_action=accept`.
98	- Claim: `overreach=unsupported` with `recommended_action=accept`.
99	- Claim: `recommended_action=filter_or_abstain` with `overreach=no_overreach`.
100	- Claim: `recommended_action=reject` while `support_source` is not `none` and `overreach=no_overreach`.
101	- Release: `abstain_required` with a confident preference reason.
102	- Release: `A_better` or `B_better` with notes s

People who liked this also viewed

6 picks

Type

human-eval-handoff-repair

24stars3forks

View on GitHub

How to install human-eval-handoff-repair?

yha9806/academic-writing-toolkit/human-eval-handoff-repair

$npx -y skills add yha9806/academic-writing-toolkit --skill human-eval-handoff-repair

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Human Eval Handoff Repair
2
3	Use this skill when the user asks to inspect, repair, migrate, or validate human-evaluation packages or filled annotation CSVs, especially when multiple package versions exist.
4
5	## First Principle
6
7	Never trust row number, filename, or visible ID alone. Treat a filled CSV as usable only after it is aligned to the target public package by stable task-specific keys and its editable labels pass schema checks.
8
9	Do not fix labels by guessing. If a row cannot be matched safely, leave its annotation fields blank and produce a refill/import template plus an unmatched reference file.
10
11	## Inputs To Locate
12
13	- Target public handoff package folder or ZIP.
14	- Filled CSVs from annotators.
15	- Any older package claimed as the source version.
16	- Current task schemas from the target UI data or target task CSVs.
17	- If relevant, QC reports from the target package.
18
19	Prefer a user-specified output directory. If none is specified, write generated reports and repaired files under `codex_outputs/` in the current workspace. Avoid synced personal document folders unless the user explicitly asks for them.
20
21	## Task Types And Stable Keys
22
23	Use these stable keys before copying any labels:
24
25	- Claim Warrant Audit: `item_id`, `tradition`, `medium`, `period`, `layer`, `dimension_id`, `claim_text`.
26	- Release Suitability Audit: `item_id`, `source_bucket`, `pipeline_mode`, `option_a_source`, `option_b_source`, `governed_option`, `option_a_text`, `option_b_text`.
27	- Card Faithfulness Spot Audit: `audit_field_id`, `tradition`, `card_type`, `dimension_id`, `field_name`, `field_value`, `source_reference_excerpt`.
28
29	If the exact stable key does not match, do not formally aggregate the row. Text-only fuzzy candidates may be generated for manual review, but must not be treated as validated mappings.
30
31	## Editable Label Columns
32
33	Claim Warrant Audit editable columns:
34
35	- `support_source`
36	- `overreach`
37	- `certainty_calibration`
38	- `recommended_action`
39	- `annotator_confidence`
40	- `notes`
41
42	Release Suitability Audit editable columns:
43
44	- `suitability_decision`
45	- `reason_primary`
46	- `risk_in_worse_option`
47	- `annotator_confidence`
48	- `notes`
49
50	Card Faithfulness Spot Audit editable columns:
51
52	- `field_status`
53	- `field_provenance`
54	- `notes`
55
56	Only copy editable label columns. Never copy old evidence text, image paths, metadata fields, or UI-only columns into a new package.
57
58	## Label Schema Checks
59
60	Claim legal values:
61
62	- `support_source`: `visual_observation`, `metadata_context`, `card_or_expert_context`, `b0_excerpt`, `mixed`, `none`
63	- `overreach`: `no_overreach`, `minor_overreach`, `major_overreach`, `unsupported`
64	- `certainty_calibration`: `well_calibrated`, `slightly_overstated`, `clearly_overstated`, `uncertain_or_not_checkable`
65	- `recommended_action`: `accept`, `revise`, `reject`, `filter_or_abstain`
66	- `annotator_confidence`: usually `1` to `5`
67
68	Release legal values:
69
70	- `suitability_decision`: `A_better`, `B_better`, `tie_both_ok`, `tie_both_bad`, `abstain_required`
71	- `reason_primary`: target package enum only; validate from the target CSV or UI schema.
72	- `risk_in_worse_option`: target package enum only; validate from the target CSV or UI schema.
73	- `annotator_confidence`: usually `1` to `5`
74
75	Card legal values:
76
77	- `field_status`: `faithful`, `over_specific`, `unsupported`, `wrong`, `unclear`
78	- `field_provenance`: `expert_critique`, `metadata`, `visual_observation`, `accepted_domain_context`, `none`, `unclear`
79
80	Normalize harmless spelling and casing only when the mapping is deterministic. Otherwise flag the value and leave it blank.
81
82	## Required QC Procedure
83
84	For every filled CSV:
85
86	1. Count rows against the target task row count.
87	2. Check required label columns are present.
88	3. Check all non-empty labels are legal enum values.
89	4. Compare non-label/source columns with the target blank CSV.
90	5. Compute exact stable-key overlap with the target package.
91	6. Identify whether the filled CSV matches the current package, an older package, or neither.
92	7. Run simple logic checks.
93	8. Produce a QC JSON and Markdown summary.
94
95	Important logic flags:
96
97	- Claim: `support_source=none` with `recommended_action=accept`.
98	- Claim: `overreach=unsupported` with `recommended_action=accept`.
99	- Claim: `recommended_action=filter_or_abstain` with `overreach=no_overreach`.
100	- Claim: `recommended_action=reject` while `support_source` is not `none` and `overreach=no_overreach`.
101	- Release: `abstain_required` with a confident preference reason.
102	- Release: `A_better` or `B_better` with notes s

People who liked this also viewed

6 picks

Type