ai-eval-ci

Name: ai-eval-ci
Author: TerminalSkills

5installs13forks

How to install ai-eval-ci?

terminalskills/skills/ai-eval-ci

$npx -y skills add TerminalSkills/skills --skill ai-eval-ci

Installs into the current project.

Summary

Run AI agent and LLM evaluations in CI/CD pipelines — automated quality gates that fail the build when AI output quality drops. Use when someone asks to "test my AI agent", "add evals to CI", "catch prompt regressions", "compare models", "evaluate LLM output quality", "set up AI

Files · 1

View on GitHub

SKILL.md

1	# AI Eval in CI
2
3	## Overview
4
5	Test AI agents and LLM outputs the same way you test code — automated evaluations that run in CI, compare against baselines, and fail the build when quality drops. No dashboards to check manually. Just `npx eval run --ci` and a red or green build.
6
7	## When to Use
8
9	- Adding quality gates before deploying AI features to production
10	- Catching prompt regressions when system prompts or models change
11	- Comparing model performance (GPT-4o vs Claude Sonnet vs local Llama)
12	- Validating RAG pipeline accuracy against a test dataset
13	- Benchmarking agent tool-calling accuracy and latency
14
15	## Instructions
16
17	### Strategy 1: Promptfoo (Config-Driven Evals)
18
19	Promptfoo is the most popular open-source eval framework. Define test cases in YAML, run against multiple providers, get a comparison matrix.
20
21	```yaml
22	# promptfooconfig.yaml — Eval configuration
23	# Tests a customer support agent across 3 models with quality assertions
24	description: "Customer support agent eval"
25
26	providers:
27	- id: openai:gpt-4o
28	- id: anthropic:messages:claude-sonnet-4-20250514
29	- id: ollama:llama3.1:8b
30
31	prompts:
32	- \|
33	You are a customer support agent for a SaaS product.
34	Respond helpfully and accurately. If you don't know, say so.
35
36	Customer message: {{message}}
37
38	tests:
39	- vars:
40	message: "How do I reset my password?"
41	assert:
42	- type: llm-rubric
43	value: "Response explains the password reset process clearly"
44	- type: not-contains
45	value: "I don't know"
46	- type: latency
47	threshold: 3000 # Must respond within 3 seconds
48
49	- vars:
50	message: "Can I get a refund for my annual plan?"
51	assert:
52	- type: llm-rubric
53	value: "Response acknowledges the refund request and explains the policy"
54	- type: not-contains
55	value: "I'm an AI" # Don't break character
56
57	- vars:
58	message: "Your product deleted all my data!"
59	assert:
60	- type: llm-rubric
61	value: "Response shows empathy, takes the issue seriously, and offers next steps"
62	- type: sentiment
63	threshold: 0.3 # Must not be dismissive
64
65	- vars:
66	message: "What's the weather in Tokyo?"
67	assert:
68	- type: llm-rubric
69	value: "Response politely redirects to product-related topics"
70	- type: not-contains
71	value: "Tokyo" # Should not answer off-topic questions
72	```
73
74	```bash
75	# Run evals locally
76	npx promptfoo@latest eval
77
78	# Run in CI with threshold — exits non-zero if any test fails
79	npx promptfoo@latest eval --ci --output results.json
80
81	# Compare two prompt versions
82	npx promptfoo@latest eval --prompts prompt-v1.txt prompt-v2.txt --share
83	```
84
85	### Strategy 2: Custom Eval Framework (TypeScript)
86
87	When you need full control — custom scoring logic, database-backed test sets, domain-specific metrics.
88
89	```typescript
90	// eval.ts — Custom AI eval framework with CI integration
91	/**
92	* Runs evaluation suites against AI agents/LLMs.
93	* Each eval defines inputs, expected behavior, and scoring criteria.
94	* Exits with code 1 if any score drops below threshold.
95	*/
96	import OpenAI from "openai";
97
98	interface EvalCase {
99	name: string;
100	input: string;
101	rubric: string; // What "good" looks like
102	threshold: number; // Minimum score 0-1
103	metadata?: Record<string, unknown>;
104	}
105
106	interface EvalResult {
107	name: string;
108	score: number;
109	pass: boolean;
110	output: string;
111	reasoning: string;
112	latencyMs: number;
113	}
114
115	const openai = new OpenAI();
116
117	/**
118	* Score an AI output using LLM-as-judge.
119	* Returns a score 0-1 with reasoning.
120	*/
121	async function judge(output: string, rubric: string): Promise<{ score: number; reasoning: string }> {
122	const response = await openai.chat.completions.create({
123	model: "gpt-4o-mini", // Cheap model for judging
124	messages: [
125	{
126	role: "system",
127	content: `You are an eval judge. Score the AI output against the rubric.
128	Return JSON: {"score": 0.0-1.0, "reasoning": "brief explanation"}
129	Score 1.0 = perfect match. Score 0.0 = complete failure.`,
130	},
131	{
132	role: "user",
133	content: `Rubric: ${rubric}\n\nAI Output:\n${output}`,
134	},
135	],
136	response_format: { type: "json_object" },
137	temperature: 0, // Deterministic ju

People who liked this also viewed

6 picks

Type

ai-eval-ci

5installs13forks

View on GitHub

How to install ai-eval-ci?

terminalskills/skills/ai-eval-ci

$npx -y skills add TerminalSkills/skills --skill ai-eval-ci

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AI Eval in CI
2
3	## Overview
4
5	Test AI agents and LLM outputs the same way you test code — automated evaluations that run in CI, compare against baselines, and fail the build when quality drops. No dashboards to check manually. Just `npx eval run --ci` and a red or green build.
6
7	## When to Use
8
9	- Adding quality gates before deploying AI features to production
10	- Catching prompt regressions when system prompts or models change
11	- Comparing model performance (GPT-4o vs Claude Sonnet vs local Llama)
12	- Validating RAG pipeline accuracy against a test dataset
13	- Benchmarking agent tool-calling accuracy and latency
14
15	## Instructions
16
17	### Strategy 1: Promptfoo (Config-Driven Evals)
18
19	Promptfoo is the most popular open-source eval framework. Define test cases in YAML, run against multiple providers, get a comparison matrix.
20
21	```yaml
22	# promptfooconfig.yaml — Eval configuration
23	# Tests a customer support agent across 3 models with quality assertions
24	description: "Customer support agent eval"
25
26	providers:
27	- id: openai:gpt-4o
28	- id: anthropic:messages:claude-sonnet-4-20250514
29	- id: ollama:llama3.1:8b
30
31	prompts:
32	- \|
33	You are a customer support agent for a SaaS product.
34	Respond helpfully and accurately. If you don't know, say so.
35
36	Customer message: {{message}}
37
38	tests:
39	- vars:
40	message: "How do I reset my password?"
41	assert:
42	- type: llm-rubric
43	value: "Response explains the password reset process clearly"
44	- type: not-contains
45	value: "I don't know"
46	- type: latency
47	threshold: 3000 # Must respond within 3 seconds
48
49	- vars:
50	message: "Can I get a refund for my annual plan?"
51	assert:
52	- type: llm-rubric
53	value: "Response acknowledges the refund request and explains the policy"
54	- type: not-contains
55	value: "I'm an AI" # Don't break character
56
57	- vars:
58	message: "Your product deleted all my data!"
59	assert:
60	- type: llm-rubric
61	value: "Response shows empathy, takes the issue seriously, and offers next steps"
62	- type: sentiment
63	threshold: 0.3 # Must not be dismissive
64
65	- vars:
66	message: "What's the weather in Tokyo?"
67	assert:
68	- type: llm-rubric
69	value: "Response politely redirects to product-related topics"
70	- type: not-contains
71	value: "Tokyo" # Should not answer off-topic questions
72	```
73
74	```bash
75	# Run evals locally
76	npx promptfoo@latest eval
77
78	# Run in CI with threshold — exits non-zero if any test fails
79	npx promptfoo@latest eval --ci --output results.json
80
81	# Compare two prompt versions
82	npx promptfoo@latest eval --prompts prompt-v1.txt prompt-v2.txt --share
83	```
84
85	### Strategy 2: Custom Eval Framework (TypeScript)
86
87	When you need full control — custom scoring logic, database-backed test sets, domain-specific metrics.
88
89	```typescript
90	// eval.ts — Custom AI eval framework with CI integration
91	/**
92	* Runs evaluation suites against AI agents/LLMs.
93	* Each eval defines inputs, expected behavior, and scoring criteria.
94	* Exits with code 1 if any score drops below threshold.
95	*/
96	import OpenAI from "openai";
97
98	interface EvalCase {
99	name: string;
100	input: string;
101	rubric: string; // What "good" looks like
102	threshold: number; // Minimum score 0-1
103	metadata?: Record<string, unknown>;
104	}
105
106	interface EvalResult {
107	name: string;
108	score: number;
109	pass: boolean;
110	output: string;
111	reasoning: string;
112	latencyMs: number;
113	}
114
115	const openai = new OpenAI();
116
117	/**
118	* Score an AI output using LLM-as-judge.
119	* Returns a score 0-1 with reasoning.
120	*/
121	async function judge(output: string, rubric: string): Promise<{ score: number; reasoning: string }> {
122	const response = await openai.chat.completions.create({
123	model: "gpt-4o-mini", // Cheap model for judging
124	messages: [
125	{
126	role: "system",
127	content: `You are an eval judge. Score the AI output against the rubric.
128	Return JSON: {"score": 0.0-1.0, "reasoning": "brief explanation"}
129	Score 1.0 = perfect match. Score 0.0 = complete failure.`,
130	},
131	{
132	role: "user",
133	content: `Rubric: ${rubric}\n\nAI Output:\n${output}`,
134	},
135	],
136	response_format: { type: "json_object" },
137	temperature: 0, // Deterministic ju

People who liked this also viewed

6 picks

Type