evaluating-llms

Name: evaluating-llms
Author: ancoleman

65installs59forks

How to install evaluating-llms?

ancoleman/ai-design-components/evaluating-llms

$npx -y skills add ancoleman/ai-design-components --skill evaluating-llms

Installs into the current project.

Summary

Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks. Use when testing prompt quality, validating RAG pipelines, measuring safety (hallucinations, bias), or comparing models for production deployment.

Files · 1

View on GitHub

SKILL.md

1	# LLM Evaluation
2
3	Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety.
4
5	## When to Use This Skill
6
7	Apply this skill when:
8
9	- Testing individual prompts for correctness and formatting
10	- Validating RAG (Retrieval-Augmented Generation) pipeline quality
11	- Measuring hallucinations, bias, or toxicity in LLM outputs
12	- Comparing different models or prompt configurations (A/B testing)
13	- Running benchmark tests (MMLU, HumanEval) to assess model capabilities
14	- Setting up production monitoring for LLM applications
15	- Integrating LLM quality checks into CI/CD pipelines
16
17	Common triggers:
18	- "How do I test if my RAG system is working correctly?"
19	- "How can I measure hallucinations in LLM outputs?"
20	- "What metrics should I use to evaluate generation quality?"
21	- "How do I compare GPT-4 vs Claude for my use case?"
22	- "How do I detect bias in LLM responses?"
23
24	## Evaluation Strategy Selection
25
26	### Decision Framework: Which Evaluation Approach?
27
28	By Task Type:
29
30	\| Task Type \| Primary Approach \| Metrics \| Tools \|
31	\|-----------\|------------------\|---------\|-------\|
32	\| Classification (sentiment, intent) \| Automated metrics \| Accuracy, Precision, Recall, F1 \| scikit-learn \|
33	\| Generation (summaries, creative text) \| LLM-as-judge + automated \| BLEU, ROUGE, BERTScore, Quality rubric \| GPT-4/Claude for judging \|
34	\| Question Answering \| Exact match + semantic similarity \| EM, F1, Cosine similarity \| Custom evaluators \|
35	\| RAG Systems \| RAGAS framework \| Faithfulness, Answer/Context relevance \| RAGAS library \|
36	\| Code Generation \| Unit tests + execution \| Pass@K, Test pass rate \| HumanEval, pytest \|
37	\| Multi-step Agents \| Task completion + tool accuracy \| Success rate, Efficiency \| Custom evaluators \|
38
39	By Volume and Cost:
40
41	\| Samples \| Speed \| Cost \| Recommended Approach \|
42	\|---------\|-------\|------\|---------------------\|
43	\| 1,000+ \| Immediate \| $0 \| Automated metrics (regex, JSON validation) \|
44	\| 100-1,000 \| Minutes \| $0.01-0.10 each \| LLM-as-judge (GPT-4, Claude) \|
45	\| < 100 \| Hours \| $1-10 each \| Human evaluation (pairwise comparison) \|
46
47	Layered Approach (Recommended for Production):
48	1. Layer 1: Automated metrics for all outputs (fast, cheap)
49	2. Layer 2: LLM-as-judge for 10% sample (nuanced quality)
50	3. Layer 3: Human review for 1% edge cases (validation)
51
52	## Core Evaluation Patterns
53
54	### Unit Evaluation (Individual Prompts)
55
56	Test single prompt-response pairs for correctness.
57
58	Methods:
59	- Exact Match: Response exactly matches expected output
60	- Regex Matching: Response follows expected pattern
61	- JSON Schema Validation: Structured output validation
62	- Keyword Presence: Required terms appear in response
63	- LLM-as-Judge: Binary pass/fail using evaluation prompt
64
65	Example Use Cases:
66	- Email classification (spam/not spam)
67	- Entity extraction (dates, names, locations)
68	- JSON output formatting validation
69	- Sentiment analysis (positive/negative/neutral)
70
71	Quick Start (Python):
72	```python
73	import pytest
74	from openai import OpenAI
75
76	client = OpenAI()
77
78	def classify_sentiment(text: str) -> str:
79	response = client.chat.completions.create(
80	model="gpt-3.5-turbo",
81	messages=[
82	{"role": "system", "content": "Classify sentiment as positive, negative, or neutral. Return only the label."},
83	{"role": "user", "content": text}
84	],
85	temperature=0
86	)
87	return response.choices[0].message.content.strip().lower()
88
89	def test_positive_sentiment():
90	result = classify_sentiment("I love this product!")
91	assert result == "positive"
92	```
93
94	For complete unit evaluation examples, see `examples/python/unit_evaluation.py` and `examples/typescript/unit-evaluation.ts`.
95
96	### RAG (Retrieval-Augmented Generation) Evaluation
97
98	Evaluate RAG systems using RAGAS framework metrics.
99
100	Critical Metrics (Priority Order):
101
102	1. Faithfulness (Target: > 0.8) - MOST CRITICAL
103	- Measures: Is the answer grounded in retrieved context?
104	- Prevents hallucinations
105	- If failing: Adjust prompt to emphasize grounding, require citations
106
107	2. Answer Relevance (Target: > 0.7)
108	- Measures: How well does the answer address the query?
109	- If failing: Improve prompt instructions, add few-shot examples
110
111	3. Context Relevance (Target: > 0.7)
112	- Measures: Are retrieved chunks relevant to the query?
113	- If failing: Improve retrieval (better embeddings, hybrid search)
114
115	4. Context Precision (Target: > 0.5)
116	- Measures: Are relevant chunks ranked higher than irrelevant?
117	- If failing: Add re-ranking step to retrieval pipeline
118
119	5. Context Recall (Target: > 0.8)
120	- Measures: Are all relevant ch

People who liked this also viewed

6 picks

Type

evaluating-llms

65installs59forks

View on GitHub

How to install evaluating-llms?

ancoleman/ai-design-components/evaluating-llms

$npx -y skills add ancoleman/ai-design-components --skill evaluating-llms

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# LLM Evaluation
2
3	Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety.
4
5	## When to Use This Skill
6
7	Apply this skill when:
8
9	- Testing individual prompts for correctness and formatting
10	- Validating RAG (Retrieval-Augmented Generation) pipeline quality
11	- Measuring hallucinations, bias, or toxicity in LLM outputs
12	- Comparing different models or prompt configurations (A/B testing)
13	- Running benchmark tests (MMLU, HumanEval) to assess model capabilities
14	- Setting up production monitoring for LLM applications
15	- Integrating LLM quality checks into CI/CD pipelines
16
17	Common triggers:
18	- "How do I test if my RAG system is working correctly?"
19	- "How can I measure hallucinations in LLM outputs?"
20	- "What metrics should I use to evaluate generation quality?"
21	- "How do I compare GPT-4 vs Claude for my use case?"
22	- "How do I detect bias in LLM responses?"
23
24	## Evaluation Strategy Selection
25
26	### Decision Framework: Which Evaluation Approach?
27
28	By Task Type:
29
30	\| Task Type \| Primary Approach \| Metrics \| Tools \|
31	\|-----------\|------------------\|---------\|-------\|
32	\| Classification (sentiment, intent) \| Automated metrics \| Accuracy, Precision, Recall, F1 \| scikit-learn \|
33	\| Generation (summaries, creative text) \| LLM-as-judge + automated \| BLEU, ROUGE, BERTScore, Quality rubric \| GPT-4/Claude for judging \|
34	\| Question Answering \| Exact match + semantic similarity \| EM, F1, Cosine similarity \| Custom evaluators \|
35	\| RAG Systems \| RAGAS framework \| Faithfulness, Answer/Context relevance \| RAGAS library \|
36	\| Code Generation \| Unit tests + execution \| Pass@K, Test pass rate \| HumanEval, pytest \|
37	\| Multi-step Agents \| Task completion + tool accuracy \| Success rate, Efficiency \| Custom evaluators \|
38
39	By Volume and Cost:
40
41	\| Samples \| Speed \| Cost \| Recommended Approach \|
42	\|---------\|-------\|------\|---------------------\|
43	\| 1,000+ \| Immediate \| $0 \| Automated metrics (regex, JSON validation) \|
44	\| 100-1,000 \| Minutes \| $0.01-0.10 each \| LLM-as-judge (GPT-4, Claude) \|
45	\| < 100 \| Hours \| $1-10 each \| Human evaluation (pairwise comparison) \|
46
47	Layered Approach (Recommended for Production):
48	1. Layer 1: Automated metrics for all outputs (fast, cheap)
49	2. Layer 2: LLM-as-judge for 10% sample (nuanced quality)
50	3. Layer 3: Human review for 1% edge cases (validation)
51
52	## Core Evaluation Patterns
53
54	### Unit Evaluation (Individual Prompts)
55
56	Test single prompt-response pairs for correctness.
57
58	Methods:
59	- Exact Match: Response exactly matches expected output
60	- Regex Matching: Response follows expected pattern
61	- JSON Schema Validation: Structured output validation
62	- Keyword Presence: Required terms appear in response
63	- LLM-as-Judge: Binary pass/fail using evaluation prompt
64
65	Example Use Cases:
66	- Email classification (spam/not spam)
67	- Entity extraction (dates, names, locations)
68	- JSON output formatting validation
69	- Sentiment analysis (positive/negative/neutral)
70
71	Quick Start (Python):
72	```python
73	import pytest
74	from openai import OpenAI
75
76	client = OpenAI()
77
78	def classify_sentiment(text: str) -> str:
79	response = client.chat.completions.create(
80	model="gpt-3.5-turbo",
81	messages=[
82	{"role": "system", "content": "Classify sentiment as positive, negative, or neutral. Return only the label."},
83	{"role": "user", "content": text}
84	],
85	temperature=0
86	)
87	return response.choices[0].message.content.strip().lower()
88
89	def test_positive_sentiment():
90	result = classify_sentiment("I love this product!")
91	assert result == "positive"
92	```
93
94	For complete unit evaluation examples, see `examples/python/unit_evaluation.py` and `examples/typescript/unit-evaluation.ts`.
95
96	### RAG (Retrieval-Augmented Generation) Evaluation
97
98	Evaluate RAG systems using RAGAS framework metrics.
99
100	Critical Metrics (Priority Order):
101
102	1. Faithfulness (Target: > 0.8) - MOST CRITICAL
103	- Measures: Is the answer grounded in retrieved context?
104	- Prevents hallucinations
105	- If failing: Adjust prompt to emphasize grounding, require citations
106
107	2. Answer Relevance (Target: > 0.7)
108	- Measures: How well does the answer address the query?
109	- If failing: Improve prompt instructions, add few-shot examples
110
111	3. Context Relevance (Target: > 0.7)
112	- Measures: Are retrieved chunks relevant to the query?
113	- If failing: Improve retrieval (better embeddings, hybrid search)
114
115	4. Context Precision (Target: > 0.5)
116	- Measures: Are relevant chunks ranked higher than irrelevant?
117	- If failing: Add re-ranking step to retrieval pipeline
118
119	5. Context Recall (Target: > 0.8)
120	- Measures: Are all relevant ch

People who liked this also viewed

6 picks

Type