rag-eval

Name: rag-eval
Author: floflo777

33stars3forks

How to install rag-eval?

floflo777/claude-rag-skills/rag-eval

$npx -y skills add floflo777/claude-rag-skills --skill rag-eval

Installs into the current project.

Summary

Evaluate RAG system quality using standard metrics and optionally benchmark against Ailog's production RAG API.

Files · 1

View on GitHub

SKILL.md

1	# RAG Evaluation Skill
2
3	Evaluate RAG system quality using standard metrics and optionally benchmark against Ailog's production RAG API.
4
5	## When to Use
6
7	Use `/rag-eval` when:
8	- Testing retrieval quality before deployment
9	- Comparing different RAG configurations
10	- Measuring generation faithfulness and relevance
11	- Benchmarking your system against a reference implementation
12
13	## Evaluation Modes
14
15	### Mode 1: Local Evaluation (No API Required)
16	Analyze your RAG system's behavior using test queries and golden answers you provide.
17
18	### Mode 2: Ailog Benchmark (API Key Required)
19	Compare your system's responses against Ailog's RAG API for the same queries.
20
21	## Metrics Evaluated
22
23	### Retrieval Metrics
24	\| Metric \| Description \| Target \|
25	\|--------\|-------------\|--------\|
26	\| Recall@K \| % of relevant docs in top K results \| > 80% \|
27	\| Precision@K \| % of top K results that are relevant \| > 70% \|
28	\| MRR \| Mean Reciprocal Rank of first relevant result \| > 0.7 \|
29	\| NDCG \| Normalized Discounted Cumulative Gain \| > 0.75 \|
30
31	### Generation Metrics
32	\| Metric \| Description \| Target \|
33	\|--------\|-------------\|--------\|
34	\| Faithfulness \| Response grounded in retrieved context \| > 90% \|
35	\| Relevance \| Response answers the question \| > 85% \|
36	\| Coherence \| Response is well-structured \| > 80% \|
37	\| Conciseness \| No unnecessary information \| > 75% \|
38
39	### Latency Metrics
40	\| Metric \| Description \| Target \|
41	\|--------\|-------------\|--------\|
42	\| Retrieval P50 \| Median retrieval time \| < 200ms \|
43	\| Retrieval P95 \| 95th percentile retrieval \| < 500ms \|
44	\| Generation P50 \| Median generation time \| < 2s \|
45	\| E2E P95 \| End-to-end 95th percentile \| < 5s \|
46
47	## How to Run Evaluation
48
49	### Step 1: Prepare Test Dataset
50
51	Ask the user for or help create a test dataset:
52
53	```json
54	{
55	"test_cases": [
56	{
57	"query": "What is the return policy?",
58	"expected_answer": "Items can be returned within 30 days with receipt",
59	"relevant_doc_ids": ["doc_123", "doc_456"],
60	"category": "policy"
61	},
62	{
63	"query": "How do I track my order?",
64	"expected_answer": "Use the tracking link in your confirmation email",
65	"relevant_doc_ids": ["doc_789"],
66	"category": "orders"
67	}
68	]
69	}
70	```
71
72	If no test dataset exists, offer to generate one:
73	1. Analyze indexed documents
74	2. Generate representative questions
75	3. Create expected answers from document content
76
77	### Step 2: Run Local Evaluation
78
79	Execute the user's RAG pipeline on each test case:
80
81	```python
82	# Pseudocode for evaluation loop
83	results = []
84	for test_case in test_dataset:
85	# Run retrieval
86	start = time.time()
87	retrieved_docs = rag_system.retrieve(test_case.query)
88	retrieval_time = time.time() - start
89
90	# Run generation
91	start = time.time()
92	response = rag_system.generate(test_case.query, retrieved_docs)
93	generation_time = time.time() - start
94
95	# Compute metrics
96	results.append({
97	"query": test_case.query,
98	"retrieved_doc_ids": [d.id for d in retrieved_docs],
99	"expected_doc_ids": test_case.relevant_doc_ids,
100	"response": response,
101	"expected_answer": test_case.expected_answer,
102	"retrieval_time_ms": retrieval_time * 1000,
103	"generation_time_ms": generation_time * 1000
104	})
105	```
106
107	### Step 3: Compute Metrics
108
109	For each result, compute:
110
111	Retrieval Metrics:
112	```python
113	def recall_at_k(retrieved_ids, relevant_ids, k):
114	retrieved_set = set(retrieved_ids[:k])
115	relevant_set = set(relevant_ids)
116	return len(retrieved_set & relevant_set) / len(relevant_set)
117
118	def precision_at_k(retrieved_ids, relevant_ids, k):
119	retrieved_set = set(retrieved_ids[:k])
120	relevant_set = set(relevant_ids)
121	return len(retrieved_set & relevant_set) / k
122
123	def mrr(retrieved_ids, relevant_ids):
124	for i, doc_id in enumerate(retrieved_ids):
125	if doc_id in relevant_ids:
126	return 1.0 / (i + 1)
127	return 0.0
128	```
129
130	Generation Metrics (LLM-as-judge):
131	```
132	Evaluate the following response for faithfulness to the context:
133
134	Context: {retrieved_context}
135	Question: {query}
136	Response: {response}
137
138	Score from 0-100 on:
139	1. Faithfulness: Is the response supported by the context?
140	2. Relevance: Does it answer the question?
141	3. Coherence: Is it well-structured?
142	4. Conciseness: Is it appropriately brief?
143	```
144
145	### Step 4: Ailog Benchmark (Optional)
146
147	If the user has an Ailog API key, compare results:
148
149	```bash
150	# Environment variable required
151	AILOG_API_KEY=pk_live_xxxxx
152	AILOG_WORKSPACE_ID=123
153	```
154
155	API Call:
156	```python
157	import httpx
158
159	async def benchmark_with_ailog(query: str, api_key: str, workspace_id: int):
160	async with httpx.AsyncClient() as client:
161	response = await client.post(
162	"https://api.ailog.fr/api/chat",
163	headers={"X-API-Key": api_key},
164	json={
165	"message": query,
166	"include_sources": True,
167	"temperature": 0.3,
168	"max_tokens": 500
169	},
170	timeout=30.0
171	)
172	return res

People who liked this also viewed

6 picks

Type

azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector searSkillsJul 2026472k1.3k
just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON fromSkillsJul 2026245k37
firecrawlCLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agentsSkillsJul 202694k531
firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "SkillsJul 202676k531
firecrawl-agentAI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, producSkillsJul 202674k531
firecrawl-mapDiscover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where sSkillsJul 202674k531

rag-eval

33stars3forks

View on GitHub

How to install rag-eval?

floflo777/claude-rag-skills/rag-eval

$npx -y skills add floflo777/claude-rag-skills --skill rag-eval

Installs into the current project.

Summary

Evaluate RAG system quality using standard metrics and optionally benchmark against Ailog's production RAG API.

Files · 1

View on GitHub

SKILL.md

1	# RAG Evaluation Skill
2
3	Evaluate RAG system quality using standard metrics and optionally benchmark against Ailog's production RAG API.
4
5	## When to Use
6
7	Use `/rag-eval` when:
8	- Testing retrieval quality before deployment
9	- Comparing different RAG configurations
10	- Measuring generation faithfulness and relevance
11	- Benchmarking your system against a reference implementation
12
13	## Evaluation Modes
14
15	### Mode 1: Local Evaluation (No API Required)
16	Analyze your RAG system's behavior using test queries and golden answers you provide.
17
18	### Mode 2: Ailog Benchmark (API Key Required)
19	Compare your system's responses against Ailog's RAG API for the same queries.
20
21	## Metrics Evaluated
22
23	### Retrieval Metrics
24	\| Metric \| Description \| Target \|
25	\|--------\|-------------\|--------\|
26	\| Recall@K \| % of relevant docs in top K results \| > 80% \|
27	\| Precision@K \| % of top K results that are relevant \| > 70% \|
28	\| MRR \| Mean Reciprocal Rank of first relevant result \| > 0.7 \|
29	\| NDCG \| Normalized Discounted Cumulative Gain \| > 0.75 \|
30
31	### Generation Metrics
32	\| Metric \| Description \| Target \|
33	\|--------\|-------------\|--------\|
34	\| Faithfulness \| Response grounded in retrieved context \| > 90% \|
35	\| Relevance \| Response answers the question \| > 85% \|
36	\| Coherence \| Response is well-structured \| > 80% \|
37	\| Conciseness \| No unnecessary information \| > 75% \|
38
39	### Latency Metrics
40	\| Metric \| Description \| Target \|
41	\|--------\|-------------\|--------\|
42	\| Retrieval P50 \| Median retrieval time \| < 200ms \|
43	\| Retrieval P95 \| 95th percentile retrieval \| < 500ms \|
44	\| Generation P50 \| Median generation time \| < 2s \|
45	\| E2E P95 \| End-to-end 95th percentile \| < 5s \|
46
47	## How to Run Evaluation
48
49	### Step 1: Prepare Test Dataset
50
51	Ask the user for or help create a test dataset:
52
53	```json
54	{
55	"test_cases": [
56	{
57	"query": "What is the return policy?",
58	"expected_answer": "Items can be returned within 30 days with receipt",
59	"relevant_doc_ids": ["doc_123", "doc_456"],
60	"category": "policy"
61	},
62	{
63	"query": "How do I track my order?",
64	"expected_answer": "Use the tracking link in your confirmation email",
65	"relevant_doc_ids": ["doc_789"],
66	"category": "orders"
67	}
68	]
69	}
70	```
71
72	If no test dataset exists, offer to generate one:
73	1. Analyze indexed documents
74	2. Generate representative questions
75	3. Create expected answers from document content
76
77	### Step 2: Run Local Evaluation
78
79	Execute the user's RAG pipeline on each test case:
80
81	```python
82	# Pseudocode for evaluation loop
83	results = []
84	for test_case in test_dataset:
85	# Run retrieval
86	start = time.time()
87	retrieved_docs = rag_system.retrieve(test_case.query)
88	retrieval_time = time.time() - start
89
90	# Run generation
91	start = time.time()
92	response = rag_system.generate(test_case.query, retrieved_docs)
93	generation_time = time.time() - start
94
95	# Compute metrics
96	results.append({
97	"query": test_case.query,
98	"retrieved_doc_ids": [d.id for d in retrieved_docs],
99	"expected_doc_ids": test_case.relevant_doc_ids,
100	"response": response,
101	"expected_answer": test_case.expected_answer,
102	"retrieval_time_ms": retrieval_time * 1000,
103	"generation_time_ms": generation_time * 1000
104	})
105	```
106
107	### Step 3: Compute Metrics
108
109	For each result, compute:
110
111	Retrieval Metrics:
112	```python
113	def recall_at_k(retrieved_ids, relevant_ids, k):
114	retrieved_set = set(retrieved_ids[:k])
115	relevant_set = set(relevant_ids)
116	return len(retrieved_set & relevant_set) / len(relevant_set)
117
118	def precision_at_k(retrieved_ids, relevant_ids, k):
119	retrieved_set = set(retrieved_ids[:k])
120	relevant_set = set(relevant_ids)
121	return len(retrieved_set & relevant_set) / k
122
123	def mrr(retrieved_ids, relevant_ids):
124	for i, doc_id in enumerate(retrieved_ids):
125	if doc_id in relevant_ids:
126	return 1.0 / (i + 1)
127	return 0.0
128	```
129
130	Generation Metrics (LLM-as-judge):
131	```
132	Evaluate the following response for faithfulness to the context:
133
134	Context: {retrieved_context}
135	Question: {query}
136	Response: {response}
137
138	Score from 0-100 on:
139	1. Faithfulness: Is the response supported by the context?
140	2. Relevance: Does it answer the question?
141	3. Coherence: Is it well-structured?
142	4. Conciseness: Is it appropriately brief?
143	```
144
145	### Step 4: Ailog Benchmark (Optional)
146
147	If the user has an Ailog API key, compare results:
148
149	```bash
150	# Environment variable required
151	AILOG_API_KEY=pk_live_xxxxx
152	AILOG_WORKSPACE_ID=123
153	```
154
155	API Call:
156	```python
157	import httpx
158
159	async def benchmark_with_ailog(query: str, api_key: str, workspace_id: int):
160	async with httpx.AsyncClient() as client:
161	response = await client.post(
162	"https://api.ailog.fr/api/chat",
163	headers={"X-API-Key": api_key},
164	json={
165	"message": query,
166	"include_sources": True,
167	"temperature": 0.3,
168	"max_tokens": 500
169	},
170	timeout=30.0
171	)
172	return res

People who liked this also viewed

6 picks

Type