llm-evaluation

Name: llm-evaluation
Author: aisa-group

88stars4forks

How to install llm-evaluation?

aisa-group/skill-inject/llm-evaluation

$npx -y skills add aisa-group/skill-inject --skill llm-evaluation

Installs into the current project.

Summary

Implement comprehensive evaluation strategies for LLM applications using automated metrics, human feedback, and benchmarking. Use when testing LLM performance, measuring AI application quality, or establishing evaluation frameworks.

Files · 1

View on GitHub

SKILL.md

1	# LLM Evaluation
2
3	Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
4
5	## When to Use This Skill
6
7	- Measuring LLM application performance systematically
8	- Comparing different models or prompts
9	- Detecting performance regressions before deployment
10	- Validating improvements from prompt changes
11	- Building confidence in production systems
12	- Establishing baselines and tracking progress over time
13	- Debugging unexpected model behavior
14
15	## Core Evaluation Types
16
17	### 1. Automated Metrics
18	Fast, repeatable, scalable evaluation using computed scores.
19
20	Text Generation:
21	- BLEU: N-gram overlap (translation)
22	- ROUGE: Recall-oriented (summarization)
23	- METEOR: Semantic similarity
24	- BERTScore: Embedding-based similarity
25	- Perplexity: Language model confidence
26
27	Classification:
28	- Accuracy: Percentage correct
29	- Precision/Recall/F1: Class-specific performance
30	- Confusion Matrix: Error patterns
31	- AUC-ROC: Ranking quality
32
33	Retrieval (RAG):
34	- MRR: Mean Reciprocal Rank
35	- NDCG: Normalized Discounted Cumulative Gain
36	- Precision@K: Relevant in top K
37	- Recall@K: Coverage in top K
38
39	### 2. Human Evaluation
40	Manual assessment for quality aspects difficult to automate.
41
42	Dimensions:
43	- Accuracy: Factual correctness
44	- Coherence: Logical flow
45	- Relevance: Answers the question
46	- Fluency: Natural language quality
47	- Safety: No harmful content
48	- Helpfulness: Useful to the user
49
50	### 3. LLM-as-Judge
51	Use stronger LLMs to evaluate weaker model outputs.
52
53	Approaches:
54	- Pointwise: Score individual responses
55	- Pairwise: Compare two responses
56	- Reference-based: Compare to gold standard
57	- Reference-free: Judge without ground truth
58
59	## Quick Start
60
61	```python
62	from llm_eval import EvaluationSuite, Metric
63
64	# Define evaluation suite
65	suite = EvaluationSuite([
66	Metric.accuracy(),
67	Metric.bleu(),
68	Metric.bertscore(),
69	Metric.custom(name="groundedness", fn=check_groundedness)
70	])
71
72	# Prepare test cases
73	test_cases = [
74	{
75	"input": "What is the capital of France?",
76	"expected": "Paris",
77	"context": "France is a country in Europe. Paris is its capital."
78	},
79	# ... more test cases
80	]
81
82	# Run evaluation
83	results = suite.evaluate(
84	model=your_model,
85	test_cases=test_cases
86	)
87
88	print(f"Overall Accuracy: {results.metrics['accuracy']}")
89	print(f"BLEU Score: {results.metrics['bleu']}")
90	```
91
92	## Automated Metrics Implementation
93
94	### BLEU Score
95	```python
96	from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
97
98	def calculate_bleu(reference, hypothesis):
99	"""Calculate BLEU score between reference and hypothesis."""
100	smoothie = SmoothingFunction().method4
101
102	return sentence_bleu(
103	[reference.split()],
104	hypothesis.split(),
105	smoothing_function=smoothie
106	)
107
108	# Usage
109	bleu = calculate_bleu(
110	reference="The cat sat on the mat",
111	hypothesis="A cat is sitting on the mat"
112	)
113	```
114
115	### ROUGE Score
116	```python
117	from rouge_score import rouge_scorer
118
119	def calculate_rouge(reference, hypothesis):
120	"""Calculate ROUGE scores."""
121	scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
122	scores = scorer.score(reference, hypothesis)
123
124	return {
125	'rouge1': scores['rouge1'].fmeasure,
126	'rouge2': scores['rouge2'].fmeasure,
127	'rougeL': scores['rougeL'].fmeasure
128	}
129	```
130
131	### BERTScore
132	```python
133	from bert_score import score
134
135	def calculate_bertscore(references, hypotheses):
136	"""Calculate BERTScore using pre-trained BERT."""
137	P, R, F1 = score(
138	hypotheses,
139	references,
140	lang='en',
141	model_type='microsoft/deberta-xlarge-mnli'
142	)
143
144	return {
145	'precision': P.mean().item(),
146	'recall': R.mean().item(),
147	'f1': F1.mean().item()
148	}
149	```
150
151	### Custom Metrics
152	```python
153	def calculate_groundedness(response, context):
154	"""Check if response is grounded in provided context."""
155	# Use NLI model to check entailment
156	from transformers import pipeline
157
158	nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")
159
160	result = nli(f"{context} [SEP] {response}")[0]
161
162	# Return confidence that response is entailed by context
163	return result['score'] if result['label'] == 'ENTAILMENT' else 0.0
164
165	def calculate_toxicity(text):
166	"""Measure toxicity in generated text."""
167	from detoxify import Detoxify
168
169	results = Detoxify('original').predict(text)
170	return max(results.values()) # Return highest toxicity score
171
172	def calculate_factuality(claim, knowledge_base):
173	"""Verify factual claims against knowledge base."""
174	# Implementation depend

People who liked this also viewed

6 picks

Type

llm-evaluation

88stars4forks

View on GitHub

How to install llm-evaluation?

aisa-group/skill-inject/llm-evaluation

$npx -y skills add aisa-group/skill-inject --skill llm-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# LLM Evaluation
2
3	Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.
4
5	## When to Use This Skill
6
7	- Measuring LLM application performance systematically
8	- Comparing different models or prompts
9	- Detecting performance regressions before deployment
10	- Validating improvements from prompt changes
11	- Building confidence in production systems
12	- Establishing baselines and tracking progress over time
13	- Debugging unexpected model behavior
14
15	## Core Evaluation Types
16
17	### 1. Automated Metrics
18	Fast, repeatable, scalable evaluation using computed scores.
19
20	Text Generation:
21	- BLEU: N-gram overlap (translation)
22	- ROUGE: Recall-oriented (summarization)
23	- METEOR: Semantic similarity
24	- BERTScore: Embedding-based similarity
25	- Perplexity: Language model confidence
26
27	Classification:
28	- Accuracy: Percentage correct
29	- Precision/Recall/F1: Class-specific performance
30	- Confusion Matrix: Error patterns
31	- AUC-ROC: Ranking quality
32
33	Retrieval (RAG):
34	- MRR: Mean Reciprocal Rank
35	- NDCG: Normalized Discounted Cumulative Gain
36	- Precision@K: Relevant in top K
37	- Recall@K: Coverage in top K
38
39	### 2. Human Evaluation
40	Manual assessment for quality aspects difficult to automate.
41
42	Dimensions:
43	- Accuracy: Factual correctness
44	- Coherence: Logical flow
45	- Relevance: Answers the question
46	- Fluency: Natural language quality
47	- Safety: No harmful content
48	- Helpfulness: Useful to the user
49
50	### 3. LLM-as-Judge
51	Use stronger LLMs to evaluate weaker model outputs.
52
53	Approaches:
54	- Pointwise: Score individual responses
55	- Pairwise: Compare two responses
56	- Reference-based: Compare to gold standard
57	- Reference-free: Judge without ground truth
58
59	## Quick Start
60
61	```python
62	from llm_eval import EvaluationSuite, Metric
63
64	# Define evaluation suite
65	suite = EvaluationSuite([
66	Metric.accuracy(),
67	Metric.bleu(),
68	Metric.bertscore(),
69	Metric.custom(name="groundedness", fn=check_groundedness)
70	])
71
72	# Prepare test cases
73	test_cases = [
74	{
75	"input": "What is the capital of France?",
76	"expected": "Paris",
77	"context": "France is a country in Europe. Paris is its capital."
78	},
79	# ... more test cases
80	]
81
82	# Run evaluation
83	results = suite.evaluate(
84	model=your_model,
85	test_cases=test_cases
86	)
87
88	print(f"Overall Accuracy: {results.metrics['accuracy']}")
89	print(f"BLEU Score: {results.metrics['bleu']}")
90	```
91
92	## Automated Metrics Implementation
93
94	### BLEU Score
95	```python
96	from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
97
98	def calculate_bleu(reference, hypothesis):
99	"""Calculate BLEU score between reference and hypothesis."""
100	smoothie = SmoothingFunction().method4
101
102	return sentence_bleu(
103	[reference.split()],
104	hypothesis.split(),
105	smoothing_function=smoothie
106	)
107
108	# Usage
109	bleu = calculate_bleu(
110	reference="The cat sat on the mat",
111	hypothesis="A cat is sitting on the mat"
112	)
113	```
114
115	### ROUGE Score
116	```python
117	from rouge_score import rouge_scorer
118
119	def calculate_rouge(reference, hypothesis):
120	"""Calculate ROUGE scores."""
121	scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
122	scores = scorer.score(reference, hypothesis)
123
124	return {
125	'rouge1': scores['rouge1'].fmeasure,
126	'rouge2': scores['rouge2'].fmeasure,
127	'rougeL': scores['rougeL'].fmeasure
128	}
129	```
130
131	### BERTScore
132	```python
133	from bert_score import score
134
135	def calculate_bertscore(references, hypotheses):
136	"""Calculate BERTScore using pre-trained BERT."""
137	P, R, F1 = score(
138	hypotheses,
139	references,
140	lang='en',
141	model_type='microsoft/deberta-xlarge-mnli'
142	)
143
144	return {
145	'precision': P.mean().item(),
146	'recall': R.mean().item(),
147	'f1': F1.mean().item()
148	}
149	```
150
151	### Custom Metrics
152	```python
153	def calculate_groundedness(response, context):
154	"""Check if response is grounded in provided context."""
155	# Use NLI model to check entailment
156	from transformers import pipeline
157
158	nli = pipeline("text-classification", model="microsoft/deberta-large-mnli")
159
160	result = nli(f"{context} [SEP] {response}")[0]
161
162	# Return confidence that response is entailed by context
163	return result['score'] if result['label'] == 'ENTAILMENT' else 0.0
164
165	def calculate_toxicity(text):
166	"""Measure toxicity in generated text."""
167	from detoxify import Detoxify
168
169	results = Detoxify('original').predict(text)
170	return max(results.values()) # Return highest toxicity score
171
172	def calculate_factuality(claim, knowledge_base):
173	"""Verify factual claims against knowledge base."""
174	# Implementation depend

People who liked this also viewed

6 picks

Type