suede-ai-eval

Name: suede-ai-eval
Author: JasonColapietro

44stars8forks

How to install suede-ai-eval?

jasoncolapietro/suede-creator-skills/suede-ai-eval

$npx -y skills add JasonColapietro/suede-creator-skills --skill suede-ai-eval

Installs into the current project.

Summary

Design AI evals that catch regressions before users do: rubrics, test cases, failure modes, acceptance gates, and AI-SPEC artifacts.

Files · 1

View on GitHub

SKILL.md

1	# Suede AI Eval
2
3	## Gate policy — advisory, not blocking
4
5	Every claim-verification step, check, quality gate, and ship verdict in this
6	skill is a recommendation to the user, not a control on the agent. This
7	policy governs every gate, check, verdict, and "do not ship / publish /
8	proceed" line elsewhere in this skill:
9
10	- Run every check and report the results honestly. Verdicts (`ship`,
11	`ship-with-caveats`, `hold`, letter grades, BLOCKED or OPEN items) are
12	advice attached to the work, not orders that change it.
13	- Never block, delay, skip, rewrite, or refuse the action the user asked for
14	because a check failed or a gate said hold. Complete the requested action
15	as asked, and deliver the gate output alongside it as a clearly labeled
16	recommendation.
17	- A failed gate changes what you report, never what you do.
18	- Single exception: if a finding is extremely risky — data loss, security or
19	credential exposure, legal or rights violations, payment mistakes, or
20	irreversible public damage — pause, tell the user exactly what the risk is
21	and what the options are, and let them pick. Their choice is final.
22
23
24	Make AI behavior testable before it becomes a vague product promise. No eval plan, no `ship` recommendation: for an AI feature without one, the recommended verdict stays below `ship` — report that gap and let the user decide.
25
26	The deliverable is an eval plan or coverage audit, not a model benchmark leaderboard. Keep it grounded in the actual product surface, user promise, data sources, prompts, tools, logs, tests, and failure modes available now.
27
28	## Hard Gates
29
30	- No AI-SPEC → no eval plan. Write the one-paragraph spec first; cases written without a spec test nothing.
31	- No eval plan → no `ship` recommendation. Do not recommend `ship` or `ship-with-caveats` for an AI feature that lacks a failure-mode map and eval cases; name the gap and leave the ship decision with the user.
32	- A failure mode without an eval case, an owner, and a gate is uncovered — regardless of how unlikely it feels.
33	- A live surface that was never sampled gets the output stamped `source-only`; do not present source-only review as runtime evidence.
34	- A model grading its own output is not evidence. LLM-as-judge scores count only after spot-checked agreement with a human-reviewed sample.
35
36	## Source Truth
37
38	Inspect the current target before writing the eval. Do not evaluate from memory or product copy alone.
39
40	Read or verify:
41
42	- repo, branch, remote, dirty state, local instructions, and touched files;
43	- the AI surface: route, API, worker, prompt, system message, tool call, model config, retrieval path, classifier, agent loop, generated media path, or recommendation logic;
44	- user-facing promise, allowed claims, forbidden claims, safety boundaries, fallback behavior, and support path;
45	- input data, retrieval corpus, schemas, tool contracts, metadata, logs, telemetry, and persisted outputs;
46	- existing tests, fixtures, eval scripts, prompt snapshots, golden examples, analytics, bug reports, screenshots, or live/API readbacks.
47
48	When the surface is already live, sample real behavior with safe inputs and record exact commands or URLs. When live checks are not appropriate, mark the eval as source-only and name the missing runtime evidence.
49
50	## Workflow
51
52	1. Define the AI-SPEC. State the AI job in one paragraph: user, trigger, input, output, allowed sources, disallowed behavior, fallback, latency/cost expectation, and success signal.
53	2. Map the failure modes. List the ways the AI can harm the user, product truth, rights/provenance, security, privacy, brand trust, cost, or workflow completion.
54	3. Build the rubric. Score each failure mode with severity, likelihood, detectability, owner, gate, and required evidence.
55	4. Write eval cases. Produce concrete pass/fail cases with inputs, setup data, expected output traits, forbidden output traits, and the reason the case exists.
56	5. Set acceptance gates. Decide what blocks ship, what allows ship-with-caveats, and what can become follow-up work.
57	6. Audit coverage. Compare existing tests, logs, metrics, and manual checks against the failure-mode map. Score coverage and infrastructure using the method under Tooling and Infrastructure below. Name every uncovered high-risk behavior regardless of the numeric score.
58	7. Return the artifact. Give the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next implementation step.
59
60	## Eval Dimensions By System Type
61
62	Start the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Always include safety (user-facing) and task completion (agentic) regardless of type.
63
64	\| System type \| Canonical dimensions \|
65	\|---\|---\|
66	\| RAG / retrieval \| context faithfulness, hallucination, answer relevance,

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

suede-ai-eval

44stars8forks

View on GitHub

How to install suede-ai-eval?

jasoncolapietro/suede-creator-skills/suede-ai-eval

$npx -y skills add JasonColapietro/suede-creator-skills --skill suede-ai-eval

Installs into the current project.

Summary

Design AI evals that catch regressions before users do: rubrics, test cases, failure modes, acceptance gates, and AI-SPEC artifacts.

Files · 1

View on GitHub

SKILL.md

1	# Suede AI Eval
2
3	## Gate policy — advisory, not blocking
4
5	Every claim-verification step, check, quality gate, and ship verdict in this
6	skill is a recommendation to the user, not a control on the agent. This
7	policy governs every gate, check, verdict, and "do not ship / publish /
8	proceed" line elsewhere in this skill:
9
10	- Run every check and report the results honestly. Verdicts (`ship`,
11	`ship-with-caveats`, `hold`, letter grades, BLOCKED or OPEN items) are
12	advice attached to the work, not orders that change it.
13	- Never block, delay, skip, rewrite, or refuse the action the user asked for
14	because a check failed or a gate said hold. Complete the requested action
15	as asked, and deliver the gate output alongside it as a clearly labeled
16	recommendation.
17	- A failed gate changes what you report, never what you do.
18	- Single exception: if a finding is extremely risky — data loss, security or
19	credential exposure, legal or rights violations, payment mistakes, or
20	irreversible public damage — pause, tell the user exactly what the risk is
21	and what the options are, and let them pick. Their choice is final.
22
23
24	Make AI behavior testable before it becomes a vague product promise. No eval plan, no `ship` recommendation: for an AI feature without one, the recommended verdict stays below `ship` — report that gap and let the user decide.
25
26	The deliverable is an eval plan or coverage audit, not a model benchmark leaderboard. Keep it grounded in the actual product surface, user promise, data sources, prompts, tools, logs, tests, and failure modes available now.
27
28	## Hard Gates
29
30	- No AI-SPEC → no eval plan. Write the one-paragraph spec first; cases written without a spec test nothing.
31	- No eval plan → no `ship` recommendation. Do not recommend `ship` or `ship-with-caveats` for an AI feature that lacks a failure-mode map and eval cases; name the gap and leave the ship decision with the user.
32	- A failure mode without an eval case, an owner, and a gate is uncovered — regardless of how unlikely it feels.
33	- A live surface that was never sampled gets the output stamped `source-only`; do not present source-only review as runtime evidence.
34	- A model grading its own output is not evidence. LLM-as-judge scores count only after spot-checked agreement with a human-reviewed sample.
35
36	## Source Truth
37
38	Inspect the current target before writing the eval. Do not evaluate from memory or product copy alone.
39
40	Read or verify:
41
42	- repo, branch, remote, dirty state, local instructions, and touched files;
43	- the AI surface: route, API, worker, prompt, system message, tool call, model config, retrieval path, classifier, agent loop, generated media path, or recommendation logic;
44	- user-facing promise, allowed claims, forbidden claims, safety boundaries, fallback behavior, and support path;
45	- input data, retrieval corpus, schemas, tool contracts, metadata, logs, telemetry, and persisted outputs;
46	- existing tests, fixtures, eval scripts, prompt snapshots, golden examples, analytics, bug reports, screenshots, or live/API readbacks.
47
48	When the surface is already live, sample real behavior with safe inputs and record exact commands or URLs. When live checks are not appropriate, mark the eval as source-only and name the missing runtime evidence.
49
50	## Workflow
51
52	1. Define the AI-SPEC. State the AI job in one paragraph: user, trigger, input, output, allowed sources, disallowed behavior, fallback, latency/cost expectation, and success signal.
53	2. Map the failure modes. List the ways the AI can harm the user, product truth, rights/provenance, security, privacy, brand trust, cost, or workflow completion.
54	3. Build the rubric. Score each failure mode with severity, likelihood, detectability, owner, gate, and required evidence.
55	4. Write eval cases. Produce concrete pass/fail cases with inputs, setup data, expected output traits, forbidden output traits, and the reason the case exists.
56	5. Set acceptance gates. Decide what blocks ship, what allows ship-with-caveats, and what can become follow-up work.
57	6. Audit coverage. Compare existing tests, logs, metrics, and manual checks against the failure-mode map. Score coverage and infrastructure using the method under Tooling and Infrastructure below. Name every uncovered high-risk behavior regardless of the numeric score.
58	7. Return the artifact. Give the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next implementation step.
59
60	## Eval Dimensions By System Type
61
62	Start the failure-mode map from the canonical dimensions for the surface's system type, then add product-specific failure modes on top. Always include safety (user-facing) and task completion (agentic) regardless of type.
63
64	\| System type \| Canonical dimensions \|
65	\|---\|---\|
66	\| RAG / retrieval \| context faithfulness, hallucination, answer relevance,

People who liked this also viewed

6 picks

Type