rag-observability-evals

Name: rag-observability-evals
Author: BagelHole

42stars4forks

How to install rag-observability-evals?

bagelhole/devops-security-agent-skills/rag-observability-evals

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill rag-observability-evals

Installs into the current project.

Summary

Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.

Files · 1

View on GitHub

SKILL.md

1	# RAG Observability and Evaluations
2
3	Run retrieval-augmented generation like a measurable production system, not a black box.
4
5	## When to Use This Skill
6
7	- Deploying a RAG system to production and need quality monitoring
8	- Setting up automated evaluation pipelines for retrieval and generation
9	- Debugging hallucination or relevance regressions
10	- Building dashboards for RAG-specific golden signals
11	- Establishing quality gates for RAG pipeline changes
12
13	## Prerequisites
14
15	- RAG pipeline with instrumented retrieval and generation stages
16	- Python 3.10+ with evaluation libraries (ragas, langchain, openai)
17	- Prometheus endpoint for custom metrics export
18	- Benchmark dataset with gold-standard question/answer/source triples
19	- OpenTelemetry SDK integrated into the RAG service
20
21	## What to Measure
22
23	### Retrieval Quality
24	- Recall@k and MRR for top-k chunks
25	- Citation coverage and source freshness
26	- Embedding drift and index staleness
27
28	### Generation Quality
29	- Groundedness score (answer supported by retrieved context)
30	- Hallucination rate by route/use case
31	- Instruction adherence and format validity
32
33	### Reliability and Cost
34	- p50/p95 latency split by retrieval vs generation
35	- Token usage per stage
36	- Cache hit rate and cost per successful answer
37
38	## RAGAS Evaluation Script
39
40	```python
41	# rag_eval.py
42	"""Evaluate RAG pipeline quality using RAGAS metrics."""
43	from ragas import evaluate
44	from ragas.metrics import (
45	faithfulness,
46	answer_relevancy,
47	context_precision,
48	context_recall,
49	context_entity_recall,
50	answer_similarity,
51	)
52	from datasets import Dataset
53	import json
54	import sys
55
56	def load_eval_dataset(path: str) -> Dataset:
57	"""Load evaluation dataset with required columns."""
58	with open(path) as f:
59	data = json.load(f)
60
61	return Dataset.from_dict({
62	"question": [d["question"] for d in data],
63	"answer": [d["generated_answer"] for d in data],
64	"contexts": [d["retrieved_contexts"] for d in data],
65	"ground_truth": [d["reference_answer"] for d in data],
66	})
67
68	def run_evaluation(dataset_path: str, output_path: str):
69	"""Run full RAGAS evaluation suite."""
70	dataset = load_eval_dataset(dataset_path)
71
72	metrics = [
73	faithfulness,
74	answer_relevancy,
75	context_precision,
76	context_recall,
77	context_entity_recall,
78	answer_similarity,
79	]
80
81	results = evaluate(dataset, metrics=metrics)
82
83	# Print summary
84	print("=== RAG Evaluation Results ===")
85	for metric_name, score in results.items():
86	print(f" {metric_name}: {score:.4f}")
87
88	# Save detailed results
89	with open(output_path, "w") as f:
90	json.dump({
91	"summary": {k: float(v) for k, v in results.items()},
92	"dataset_size": len(dataset),
93	}, f, indent=2)
94
95	return results
96
97	if __name__ == "__main__":
98	run_evaluation(sys.argv[1], sys.argv[2])
99	```
100
101	## Groundedness Scoring
102
103	```python
104	# groundedness.py
105	"""Score whether generated answers are grounded in retrieved context."""
106	from openai import OpenAI
107	import json
108	from typing import List
109
110	client = OpenAI()
111
112	GROUNDEDNESS_PROMPT = """You are evaluating whether an AI answer is fully grounded
113	in the provided context documents. Score each claim in the answer.
114
115	Context documents:
116	{contexts}
117
118	Answer to evaluate:
119	{answer}
120
121	For each distinct claim in the answer, determine:
122	1. SUPPORTED - the claim is directly supported by the context
123	2. PARTIALLY_SUPPORTED - the claim is partially supported
124	3. NOT_SUPPORTED - the claim has no support in the context
125
126	Return JSON:
127	{{
128	"claims": [
129	{{"claim": "...", "verdict": "SUPPORTED\|PARTIALLY_SUPPORTED\|NOT_SUPPORTED", "evidence": "..."}}
130	],
131	"groundedness_score": <float 0-1>,
132	"unsupported_claims": ["..."]
133	}}
134	"""
135
136	def score_groundedness(answer: str, contexts: List[str]) -> dict:
137	"""Score groundedness of a single answer against its contexts."""
138	context_text = "\n---\n".join(
139	f"[Document {i+1}]: {c}" for i, c in enumerate(contexts)
140	)
141
142	response = client.chat.completions.create(
143	model="gpt-4o",
144	messages=[{
145	"role": "user",
146	"content": GROUNDEDNESS_PROMPT.format(
147	contexts=context_text, answer=answer
148	),
149	}],
150	response_format={"type": "json_object"},
151	temperature=0,
152	)
153
154	return json.loads(response.choices[0].message.content)
155
156	def batch_groundedness(eval_data: list) -> dict:
157	"""Score groundedness for a batch of QA pairs."""
158	scores = []
159	unsupported_count = 0
160	total_claims = 0
161
162	for item in eval_data:
163	result = score_groundedness(
164	item["generated_answer"],
165	item["retrieved_contexts"],
166	)
167	scores.append(result["groundedness

People who liked this also viewed

6 picks

Type

rag-observability-evals

42stars4forks

View on GitHub

How to install rag-observability-evals?

bagelhole/devops-security-agent-skills/rag-observability-evals

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill rag-observability-evals

Installs into the current project.

Summary

Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.

Files · 1

View on GitHub

SKILL.md

1	# RAG Observability and Evaluations
2
3	Run retrieval-augmented generation like a measurable production system, not a black box.
4
5	## When to Use This Skill
6
7	- Deploying a RAG system to production and need quality monitoring
8	- Setting up automated evaluation pipelines for retrieval and generation
9	- Debugging hallucination or relevance regressions
10	- Building dashboards for RAG-specific golden signals
11	- Establishing quality gates for RAG pipeline changes
12
13	## Prerequisites
14
15	- RAG pipeline with instrumented retrieval and generation stages
16	- Python 3.10+ with evaluation libraries (ragas, langchain, openai)
17	- Prometheus endpoint for custom metrics export
18	- Benchmark dataset with gold-standard question/answer/source triples
19	- OpenTelemetry SDK integrated into the RAG service
20
21	## What to Measure
22
23	### Retrieval Quality
24	- Recall@k and MRR for top-k chunks
25	- Citation coverage and source freshness
26	- Embedding drift and index staleness
27
28	### Generation Quality
29	- Groundedness score (answer supported by retrieved context)
30	- Hallucination rate by route/use case
31	- Instruction adherence and format validity
32
33	### Reliability and Cost
34	- p50/p95 latency split by retrieval vs generation
35	- Token usage per stage
36	- Cache hit rate and cost per successful answer
37
38	## RAGAS Evaluation Script
39
40	```python
41	# rag_eval.py
42	"""Evaluate RAG pipeline quality using RAGAS metrics."""
43	from ragas import evaluate
44	from ragas.metrics import (
45	faithfulness,
46	answer_relevancy,
47	context_precision,
48	context_recall,
49	context_entity_recall,
50	answer_similarity,
51	)
52	from datasets import Dataset
53	import json
54	import sys
55
56	def load_eval_dataset(path: str) -> Dataset:
57	"""Load evaluation dataset with required columns."""
58	with open(path) as f:
59	data = json.load(f)
60
61	return Dataset.from_dict({
62	"question": [d["question"] for d in data],
63	"answer": [d["generated_answer"] for d in data],
64	"contexts": [d["retrieved_contexts"] for d in data],
65	"ground_truth": [d["reference_answer"] for d in data],
66	})
67
68	def run_evaluation(dataset_path: str, output_path: str):
69	"""Run full RAGAS evaluation suite."""
70	dataset = load_eval_dataset(dataset_path)
71
72	metrics = [
73	faithfulness,
74	answer_relevancy,
75	context_precision,
76	context_recall,
77	context_entity_recall,
78	answer_similarity,
79	]
80
81	results = evaluate(dataset, metrics=metrics)
82
83	# Print summary
84	print("=== RAG Evaluation Results ===")
85	for metric_name, score in results.items():
86	print(f" {metric_name}: {score:.4f}")
87
88	# Save detailed results
89	with open(output_path, "w") as f:
90	json.dump({
91	"summary": {k: float(v) for k, v in results.items()},
92	"dataset_size": len(dataset),
93	}, f, indent=2)
94
95	return results
96
97	if __name__ == "__main__":
98	run_evaluation(sys.argv[1], sys.argv[2])
99	```
100
101	## Groundedness Scoring
102
103	```python
104	# groundedness.py
105	"""Score whether generated answers are grounded in retrieved context."""
106	from openai import OpenAI
107	import json
108	from typing import List
109
110	client = OpenAI()
111
112	GROUNDEDNESS_PROMPT = """You are evaluating whether an AI answer is fully grounded
113	in the provided context documents. Score each claim in the answer.
114
115	Context documents:
116	{contexts}
117
118	Answer to evaluate:
119	{answer}
120
121	For each distinct claim in the answer, determine:
122	1. SUPPORTED - the claim is directly supported by the context
123	2. PARTIALLY_SUPPORTED - the claim is partially supported
124	3. NOT_SUPPORTED - the claim has no support in the context
125
126	Return JSON:
127	{{
128	"claims": [
129	{{"claim": "...", "verdict": "SUPPORTED\|PARTIALLY_SUPPORTED\|NOT_SUPPORTED", "evidence": "..."}}
130	],
131	"groundedness_score": <float 0-1>,
132	"unsupported_claims": ["..."]
133	}}
134	"""
135
136	def score_groundedness(answer: str, contexts: List[str]) -> dict:
137	"""Score groundedness of a single answer against its contexts."""
138	context_text = "\n---\n".join(
139	f"[Document {i+1}]: {c}" for i, c in enumerate(contexts)
140	)
141
142	response = client.chat.completions.create(
143	model="gpt-4o",
144	messages=[{
145	"role": "user",
146	"content": GROUNDEDNESS_PROMPT.format(
147	contexts=context_text, answer=answer
148	),
149	}],
150	response_format={"type": "json_object"},
151	temperature=0,
152	)
153
154	return json.loads(response.choices[0].message.content)
155
156	def batch_groundedness(eval_data: list) -> dict:
157	"""Score groundedness for a batch of QA pairs."""
158	scores = []
159	unsupported_count = 0
160	total_claims = 0
161
162	for item in eval_data:
163	result = score_groundedness(
164	item["generated_answer"],
165	item["retrieved_contexts"],
166	)
167	scores.append(result["groundedness

People who liked this also viewed

6 picks

Type