verification-gate-fail-closed

Name: verification-gate-fail-closed
Author: Soul-Brews-Studio

109stars55forks

How to install verification-gate-fail-closed?

soul-brews-studio/arra-oracle-skills-cli/verification-gate-fail-closed

$npx -y skills add Soul-Brews-Studio/arra-oracle-skills-cli --skill verification-gate-fail-closed

Installs into the current project.

Summary

Reference for building verification gates with fail-closed design - when verification tools fail, mark results as UNVERIFIED and report clearly, never silent pass-through

Files · 1

View on GitHub

SKILL.md

1	# Verification Gate: Fail-Closed Design
2
3	> "A gate that breaks silently is worse than no gate."
4
5	## Overview
6
7	A verification gate that silently passes when the verification tool breaks is not a gate—it's a security vulnerability.
8
9	Core principle: Verification tools can fail. When they do, you must mark results as UNVERIFIED and report the failure clearly. Silent pass-through creates false confidence.
10
11	Violating the letter of fail-closed design is violating the spirit of safety.
12
13	## When to Use
14
15	- Building orchestrators, pipelines, or gates that check work before marking it complete
16	- Integrating verification tools (OCR validators, cross-model checkers, fact-check services)
17	- Any place where "tool passed" becomes a claim in a manifest, report, or downstream decision
18	- When tool failure (network error, timeout, API limit, parsing failure) is possible
19	- Marking batches or manifests as "verified" based on tool output
20
21	When NOT to use:
22	- Simple linear validation (if [check], then [proceed]) - use error handling instead
23	- Verification where tool failure causes immediate hard stop (use try/catch + propagate error)
24	- Cases where unverified results cannot flow downstream
25
26	---
27
28	## The Gate Function (5 Steps)
29
30	```
31	BEFORE marking any output as "verified" or "passed":
32
33	1. INVOKE: Call verification tool with full context
34	2. WAIT: Let tool finish completely (no timeouts before completion)
35	3. PARSE: Extract result AND error codes/messages (not just final status)
36	4. DECIDE:
37	- If tool succeeded: Mark as VERIFIED + record evidence
38	- If tool failed: Mark as UNVERIFIED + record failure reason
39	5. REPORT: Always distinguish verified/unverified/skipped/failed in output
40
41	Skip or hide any step = hidden vulnerability, not efficiency
42	```
43
44	---
45
46	## Red Flags — STOP
47
48	Watch for these patterns when designing gates. These are errors:
49
50	- Tool error → gate returns `success: true` (silent fail-through)
51	- Tool timeout → skip verification, proceed anyway
52	- Network error → assume tool would have passed
53	- Tool returns partial results → treat as full verification
54	- Agent reports "success" → trust without checking tool status
55	- Manifest counts skipped items as "verified"
56	- Any code path where tool error doesn't set result to UNVERIFIED
57	- "Tool probably would pass if it ran correctly"
58	- Proceeding when verification status is uncertain
59
60	---
61
62	## Rationalization Prevention
63
64	Real excuses Oracles have made (and why they're wrong):
65
66	\| Excuse \| Reality \| Oracle Lesson \|
67	\|--------\|---------\|---------------\|
68	\| "Tool error doesn't mean data is wrong" \| Correct. But YOU don't know if it's right either. Mark UNVERIFIED. \| Ink translate_book.sh:75: agy crashed → Ink kept raw file → reported as verified \|
69	\| "We can skip verification for this batch" \| Verification exists to catch what you can't see. Skipping defeats the purpose. \| OCR 94% confident → team skipped manual review → shipped 80% wrong \|
70	\| "Subagent reported success" \| Subagent ≠ verification tool. If tool actually failed, agent's report is fiction. \| earthPGS mining: agent said "all verified" → tool logs showed 40% failed \|
71	\| "Tool was flaky, so this might pass next time" \| "Might" ≠ "is verified". Mark UNVERIFIED. \| Vid proof1 fail-closed: must mark false-positive as UNVERIFIED not "probably works" \|
72	\| "Partial screenshot covered the key part" \| You didn't verify the whole output. You verified what you SAW. Mark that scope. \| Evidence before claims: only mark what you actually checked \|
73	\| "Manifest says verified, so assume rest are too" \| Manifest is a claim. Check the tool logs. \| CSO lesson: description of tool ≠ tool actually running \|
74	\| "Different gate tool, different standard" \| All gates must fail-closed. Standard is: tool failure = UNVERIFIED. \| Pagis fact-check gate, Ink OCR gate, Vid proof gate: same law applies \|
75
76	---
77
78	## Oracle-Specific Implementation
79
80	### UNVERIFIED Marking
81
82	When verification tool fails (error, timeout, crash, API blocked, parse failure):
83
84	```
85	Record result as:
86	{
87	status: "UNVERIFIED",
88	reason: "[specific failure: 'agy timeout 60s' \| 'network 429' \| 'parse error row 42' \| 'tool crashed']",
89	attempted: true,
90	tool_exit_code: [actual code or error],
91	evidence: [what we were able to check before failure]
92	}
93	```
94
95	Never:
96	```
97	# BAD: silently passes
98	return success if tool_error_code exists
99
100	# BAD: assumes tool would pass
101	if network_error then continue
102	```
103
104	### Manifest Honesty
105
106	Manifest must distinguish three categories:
107
108	```
109	total_items: 100
110	verified: 42 # Tool completed, returned pass
111	unverified: 18 # Tool failed/timed out/partial
112	skipped: 40 # Deliberately skipped (not a gate failure)
113	failed: 0 # Tool ran, explicitly marked as failed
114
115	# Report ONLY:
116	passed_percentage = verified / total_items

People who liked this also viewed

6 picks

Type

verification-gate-fail-closed

109stars55forks

View on GitHub

How to install verification-gate-fail-closed?

soul-brews-studio/arra-oracle-skills-cli/verification-gate-fail-closed

$npx -y skills add Soul-Brews-Studio/arra-oracle-skills-cli --skill verification-gate-fail-closed

Installs into the current project.

Summary

Reference for building verification gates with fail-closed design - when verification tools fail, mark results as UNVERIFIED and report clearly, never silent pass-through

Files · 1

View on GitHub

SKILL.md

1	# Verification Gate: Fail-Closed Design
2
3	> "A gate that breaks silently is worse than no gate."
4
5	## Overview
6
7	A verification gate that silently passes when the verification tool breaks is not a gate—it's a security vulnerability.
8
9	Core principle: Verification tools can fail. When they do, you must mark results as UNVERIFIED and report the failure clearly. Silent pass-through creates false confidence.
10
11	Violating the letter of fail-closed design is violating the spirit of safety.
12
13	## When to Use
14
15	- Building orchestrators, pipelines, or gates that check work before marking it complete
16	- Integrating verification tools (OCR validators, cross-model checkers, fact-check services)
17	- Any place where "tool passed" becomes a claim in a manifest, report, or downstream decision
18	- When tool failure (network error, timeout, API limit, parsing failure) is possible
19	- Marking batches or manifests as "verified" based on tool output
20
21	When NOT to use:
22	- Simple linear validation (if [check], then [proceed]) - use error handling instead
23	- Verification where tool failure causes immediate hard stop (use try/catch + propagate error)
24	- Cases where unverified results cannot flow downstream
25
26	---
27
28	## The Gate Function (5 Steps)
29
30	```
31	BEFORE marking any output as "verified" or "passed":
32
33	1. INVOKE: Call verification tool with full context
34	2. WAIT: Let tool finish completely (no timeouts before completion)
35	3. PARSE: Extract result AND error codes/messages (not just final status)
36	4. DECIDE:
37	- If tool succeeded: Mark as VERIFIED + record evidence
38	- If tool failed: Mark as UNVERIFIED + record failure reason
39	5. REPORT: Always distinguish verified/unverified/skipped/failed in output
40
41	Skip or hide any step = hidden vulnerability, not efficiency
42	```
43
44	---
45
46	## Red Flags — STOP
47
48	Watch for these patterns when designing gates. These are errors:
49
50	- Tool error → gate returns `success: true` (silent fail-through)
51	- Tool timeout → skip verification, proceed anyway
52	- Network error → assume tool would have passed
53	- Tool returns partial results → treat as full verification
54	- Agent reports "success" → trust without checking tool status
55	- Manifest counts skipped items as "verified"
56	- Any code path where tool error doesn't set result to UNVERIFIED
57	- "Tool probably would pass if it ran correctly"
58	- Proceeding when verification status is uncertain
59
60	---
61
62	## Rationalization Prevention
63
64	Real excuses Oracles have made (and why they're wrong):
65
66	\| Excuse \| Reality \| Oracle Lesson \|
67	\|--------\|---------\|---------------\|
68	\| "Tool error doesn't mean data is wrong" \| Correct. But YOU don't know if it's right either. Mark UNVERIFIED. \| Ink translate_book.sh:75: agy crashed → Ink kept raw file → reported as verified \|
69	\| "We can skip verification for this batch" \| Verification exists to catch what you can't see. Skipping defeats the purpose. \| OCR 94% confident → team skipped manual review → shipped 80% wrong \|
70	\| "Subagent reported success" \| Subagent ≠ verification tool. If tool actually failed, agent's report is fiction. \| earthPGS mining: agent said "all verified" → tool logs showed 40% failed \|
71	\| "Tool was flaky, so this might pass next time" \| "Might" ≠ "is verified". Mark UNVERIFIED. \| Vid proof1 fail-closed: must mark false-positive as UNVERIFIED not "probably works" \|
72	\| "Partial screenshot covered the key part" \| You didn't verify the whole output. You verified what you SAW. Mark that scope. \| Evidence before claims: only mark what you actually checked \|
73	\| "Manifest says verified, so assume rest are too" \| Manifest is a claim. Check the tool logs. \| CSO lesson: description of tool ≠ tool actually running \|
74	\| "Different gate tool, different standard" \| All gates must fail-closed. Standard is: tool failure = UNVERIFIED. \| Pagis fact-check gate, Ink OCR gate, Vid proof gate: same law applies \|
75
76	---
77
78	## Oracle-Specific Implementation
79
80	### UNVERIFIED Marking
81
82	When verification tool fails (error, timeout, crash, API blocked, parse failure):
83
84	```
85	Record result as:
86	{
87	status: "UNVERIFIED",
88	reason: "[specific failure: 'agy timeout 60s' \| 'network 429' \| 'parse error row 42' \| 'tool crashed']",
89	attempted: true,
90	tool_exit_code: [actual code or error],
91	evidence: [what we were able to check before failure]
92	}
93	```
94
95	Never:
96	```
97	# BAD: silently passes
98	return success if tool_error_code exists
99
100	# BAD: assumes tool would pass
101	if network_error then continue
102	```
103
104	### Manifest Honesty
105
106	Manifest must distinguish three categories:
107
108	```
109	total_items: 100
110	verified: 42 # Tool completed, returned pass
111	unverified: 18 # Tool failed/timed out/partial
112	skipped: 40 # Deliberately skipped (not a gate failure)
113	failed: 0 # Tool ran, explicitly marked as failed
114
115	# Report ONLY:
116	passed_percentage = verified / total_items

People who liked this also viewed

6 picks

Type