evaluate

Name: evaluate
Author: softspark

161stars21forks

How to install evaluate?

softspark/ai-toolkit/evaluate

$npx -y skills add softspark/ai-toolkit --skill evaluate

Installs into the current project.

Summary

Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.

Files · 1

View on GitHub

SKILL.md

1	# RAG Evaluation
2
3	Evaluate RAG quality using LLM-as-a-Judge methodology.
4
5	## Usage
6
7	```
8	/evaluate [--threshold 0.7]
9	```
10
11	## Execution
12
13	### Direct Execution (recommended for most projects)
14	```bash
15	# Run RAG evaluation
16	python3 scripts/evaluate_rag.py
17
18	# With custom thresholds
19	python3 scripts/evaluate_rag.py \
20	--faithfulness 0.7 \
21	--relevancy 0.7 \
22	--context 0.6
23
24	# Detect knowledge gaps
25	python3 scripts/knowledge_gaps.py --detect
26
27	# Generate gap report
28	python3 scripts/knowledge_gaps.py --report
29	```
30
31	### Docker Execution (containerized projects)
32	```bash
33	# Replace {api-container} with your API server container name
34	docker exec {api-container} python3 scripts/evaluate_rag.py
35
36	# With custom thresholds
37	docker exec {api-container} python3 scripts/evaluate_rag.py \
38	--faithfulness 0.7 \
39	--relevancy 0.7 \
40	--context 0.6
41
42	# Detect knowledge gaps
43	docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
44
45	# Generate gap report
46	docker exec {api-container} python3 scripts/knowledge_gaps.py --report
47	```
48
49	## Metrics
50
51	\| Metric \| Description \| Target \|
52	\|--------\|-------------\|--------\|
53	\| Faithfulness \| Is answer based on context? \| >70% \|
54	\| Relevancy \| Does answer address question? \| >70% \|
55	\| Context Precision \| Is found context accurate? \| >60% \|
56
57	## Evaluation Process
58
59	1. Generate test queries from golden dataset
60	2. Execute RAG pipeline for each query
61	3. LLM judges each response on metrics
62	4. Report aggregate scores
63
64	## Golden Dataset
65
66	Located at: `scripts/golden_dataset.json` (or project-specific path)
67
68	```json
69	{
70	"queries": [
71	{
72	"query": "How to configure rate limiting?",
73	"expected_topics": ["nginx", "rate-limiting"],
74	"expected_sources": ["kb/nginx/howto/rate-limiting.md"]
75	}
76	]
77	}
78	```
79
80	## Output Example
81
82	```
83	RAG Evaluation Results
84	======================
85	Total Queries: 50
86	Average Faithfulness: 0.82
87	Average Relevancy: 0.78
88	Average Context Precision: 0.71
89
90	Quality: GOOD
91
92	Failed Queries (faithfulness < 0.7):
93	- Query: "How to backup PostgreSQL?"
94	Score: 0.45
95	Issue: No relevant documents found
96	```
97
98	## Knowledge Gaps
99
100	After evaluation, check for gaps:
101
102	```bash
103	# Direct execution
104	python3 scripts/knowledge_gaps.py --detect
105
106	# Docker execution
107	docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
108	```
109
110	Output:
111	```
112	Knowledge Gaps Detected:
113	1. PostgreSQL backup procedures (5 failed queries)
114	2. Redis caching configuration (3 failed queries)
115	3. Ollama model selection (2 failed queries)
116	```
117
118	## Quality Gates
119
120	- [ ] Faithfulness >70%
121	- [ ] Relevancy >70%
122	- [ ] Context Precision >60%
123	- [ ] No critical knowledge gaps
124
125	## Rules
126
127	- MUST use a golden dataset — never evaluate on synthetic queries only
128	- NEVER report a score without listing the failed queries alongside it
129	- CRITICAL: if the golden dataset is missing, stop and ask the user to provide one
130	- MANDATORY: thresholds come from project config, not hardcoded defaults, when available
131
132	## Gotchas
133
134	- LLM-as-a-judge scores are non-deterministic; a single run fluctuates by ±10 points even with `temperature=0`. Always report the average and stddev over ≥3 runs, not a one-shot number.
135	- The default threshold trio (0.7 / 0.7 / 0.6) was calibrated on English KBs. Multilingual corpora (Polish + English in the same index) score systematically 5-15 points lower — recalibrate per language, or split the golden dataset by language.
136	- Golden datasets drift: when the KB is reindexed or documents are renamed, `expected_sources` may point at moved or deleted paths. A sudden drop in `context_precision` across unrelated queries usually means dataset rot, not RAG regression — validate the dataset paths first.
137	- Judges often reward verbose answers as "more faithful" because there is more text to ground. Tune the judge prompt to penalize padding, or cap answer length in the generator before evaluation.
138
139	## When NOT to Use
140
141	- For auditing skill quality (the 5-criteria check) — that lives in `scripts/evaluate_skills.py`
142	- For general-purpose LLM output scoring without a KB — use `/review` or a tailored prompt
143	- For unit tests or code correctness — use `/test`
144	- For continuous evaluation without a golden dataset — build the dataset first

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

evaluate

161stars21forks

View on GitHub

How to install evaluate?

softspark/ai-toolkit/evaluate

$npx -y skills add softspark/ai-toolkit --skill evaluate

Installs into the current project.

Summary

Evaluates RAG retrieval and LLM-as-judge metrics (faithfulness, relevancy, context precision). Triggers: measure RAG quality, knowledge gap, RAG eval, golden dataset.

Files · 1

View on GitHub

SKILL.md

1	# RAG Evaluation
2
3	Evaluate RAG quality using LLM-as-a-Judge methodology.
4
5	## Usage
6
7	```
8	/evaluate [--threshold 0.7]
9	```
10
11	## Execution
12
13	### Direct Execution (recommended for most projects)
14	```bash
15	# Run RAG evaluation
16	python3 scripts/evaluate_rag.py
17
18	# With custom thresholds
19	python3 scripts/evaluate_rag.py \
20	--faithfulness 0.7 \
21	--relevancy 0.7 \
22	--context 0.6
23
24	# Detect knowledge gaps
25	python3 scripts/knowledge_gaps.py --detect
26
27	# Generate gap report
28	python3 scripts/knowledge_gaps.py --report
29	```
30
31	### Docker Execution (containerized projects)
32	```bash
33	# Replace {api-container} with your API server container name
34	docker exec {api-container} python3 scripts/evaluate_rag.py
35
36	# With custom thresholds
37	docker exec {api-container} python3 scripts/evaluate_rag.py \
38	--faithfulness 0.7 \
39	--relevancy 0.7 \
40	--context 0.6
41
42	# Detect knowledge gaps
43	docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
44
45	# Generate gap report
46	docker exec {api-container} python3 scripts/knowledge_gaps.py --report
47	```
48
49	## Metrics
50
51	\| Metric \| Description \| Target \|
52	\|--------\|-------------\|--------\|
53	\| Faithfulness \| Is answer based on context? \| >70% \|
54	\| Relevancy \| Does answer address question? \| >70% \|
55	\| Context Precision \| Is found context accurate? \| >60% \|
56
57	## Evaluation Process
58
59	1. Generate test queries from golden dataset
60	2. Execute RAG pipeline for each query
61	3. LLM judges each response on metrics
62	4. Report aggregate scores
63
64	## Golden Dataset
65
66	Located at: `scripts/golden_dataset.json` (or project-specific path)
67
68	```json
69	{
70	"queries": [
71	{
72	"query": "How to configure rate limiting?",
73	"expected_topics": ["nginx", "rate-limiting"],
74	"expected_sources": ["kb/nginx/howto/rate-limiting.md"]
75	}
76	]
77	}
78	```
79
80	## Output Example
81
82	```
83	RAG Evaluation Results
84	======================
85	Total Queries: 50
86	Average Faithfulness: 0.82
87	Average Relevancy: 0.78
88	Average Context Precision: 0.71
89
90	Quality: GOOD
91
92	Failed Queries (faithfulness < 0.7):
93	- Query: "How to backup PostgreSQL?"
94	Score: 0.45
95	Issue: No relevant documents found
96	```
97
98	## Knowledge Gaps
99
100	After evaluation, check for gaps:
101
102	```bash
103	# Direct execution
104	python3 scripts/knowledge_gaps.py --detect
105
106	# Docker execution
107	docker exec {api-container} python3 scripts/knowledge_gaps.py --detect
108	```
109
110	Output:
111	```
112	Knowledge Gaps Detected:
113	1. PostgreSQL backup procedures (5 failed queries)
114	2. Redis caching configuration (3 failed queries)
115	3. Ollama model selection (2 failed queries)
116	```
117
118	## Quality Gates
119
120	- [ ] Faithfulness >70%
121	- [ ] Relevancy >70%
122	- [ ] Context Precision >60%
123	- [ ] No critical knowledge gaps
124
125	## Rules
126
127	- MUST use a golden dataset — never evaluate on synthetic queries only
128	- NEVER report a score without listing the failed queries alongside it
129	- CRITICAL: if the golden dataset is missing, stop and ask the user to provide one
130	- MANDATORY: thresholds come from project config, not hardcoded defaults, when available
131
132	## Gotchas
133
134	- LLM-as-a-judge scores are non-deterministic; a single run fluctuates by ±10 points even with `temperature=0`. Always report the average and stddev over ≥3 runs, not a one-shot number.
135	- The default threshold trio (0.7 / 0.7 / 0.6) was calibrated on English KBs. Multilingual corpora (Polish + English in the same index) score systematically 5-15 points lower — recalibrate per language, or split the golden dataset by language.
136	- Golden datasets drift: when the KB is reindexed or documents are renamed, `expected_sources` may point at moved or deleted paths. A sudden drop in `context_precision` across unrelated queries usually means dataset rot, not RAG regression — validate the dataset paths first.
137	- Judges often reward verbose answers as "more faithful" because there is more text to ground. Tune the judge prompt to penalize padding, or cap answer length in the generator before evaluation.
138
139	## When NOT to Use
140
141	- For auditing skill quality (the 5-criteria check) — that lives in `scripts/evaluate_skills.py`
142	- For general-purpose LLM output scoring without a KB — use `/review` or a tailored prompt
143	- For unit tests or code correctness — use `/test`
144	- For continuous evaluation without a golden dataset — build the dataset first

People who liked this also viewed

6 picks

Type