rag-audit

Name: rag-audit
Author: floflo777

33stars3forks

How to install rag-audit?

floflo777/claude-rag-skills/rag-audit

$npx -y skills add floflo777/claude-rag-skills --skill rag-audit

Installs into the current project.

Summary

Analyze RAG (Retrieval-Augmented Generation) implementations for anti-patterns, performance issues, and best practices violations.

Files · 1

View on GitHub

SKILL.md

1	# RAG Audit Skill
2
3	Analyze RAG (Retrieval-Augmented Generation) implementations for anti-patterns, performance issues, and best practices violations.
4
5	## When to Use
6
7	Use `/rag-audit` when:
8	- Reviewing existing RAG code for quality issues
9	- Before deploying a RAG system to production
10	- Debugging retrieval or generation problems
11	- Optimizing RAG pipeline performance
12
13	## What This Skill Does
14
15	1. Code Analysis: Scans your codebase for RAG-related code (embeddings, vector stores, retrieval, generation)
16	2. Anti-Pattern Detection: Identifies common mistakes and suboptimal patterns
17	3. Best Practices Check: Validates against industry standards
18	4. Recommendations: Provides actionable fixes with code examples
19
20	## Audit Categories
21
22	### 1. Chunking Strategy
23	- [ ] Chunk size appropriateness (too large loses precision, too small loses context)
24	- [ ] Overlap configuration (recommended: 10-20% of chunk size)
25	- [ ] Document-type specific chunking (code vs prose vs tables)
26	- [ ] Metadata preservation during chunking
27
28	### 2. Embedding Configuration
29	- [ ] Model selection for use case (multilingual, code, general)
30	- [ ] Dimension efficiency (smaller dims for speed, larger for accuracy)
31	- [ ] Batch processing for large document sets
32	- [ ] Embedding caching to avoid recomputation
33
34	### 3. Vector Store Setup
35	- [ ] Index type selection (HNSW vs IVF vs flat)
36	- [ ] Distance metric matching (cosine for normalized, L2 for raw)
37	- [ ] Collection/namespace organization
38	- [ ] Metadata filtering capabilities
39
40	### 4. Retrieval Pipeline
41	- [ ] Top-k selection (too few misses context, too many adds noise)
42	- [ ] Score thresholding implementation
43	- [ ] Hybrid search (dense + sparse/BM25)
44	- [ ] Reranking stage presence
45	- [ ] Query expansion/transformation
46
47	### 5. Generation Configuration
48	- [ ] Context window utilization
49	- [ ] System prompt quality
50	- [ ] Source citation implementation
51	- [ ] Hallucination guardrails
52	- [ ] Temperature settings for factual tasks
53
54	### 6. Production Readiness
55	- [ ] Error handling and fallbacks
56	- [ ] Logging and observability
57	- [ ] Rate limiting and caching
58	- [ ] Cost optimization (model selection, caching)
59
60	## How to Run an Audit
61
62	When the user invokes `/rag-audit`, follow this process:
63
64	### Step 1: Discover RAG Code
65	Search for RAG-related patterns in the codebase:
66
67	```
68	Patterns to search:
69	- "embedding" OR "embeddings" OR "embed("
70	- "vector" OR "vectorstore" OR "vector_store"
71	- "qdrant" OR "pinecone" OR "chroma" OR "weaviate" OR "milvus"
72	- "chunk" OR "chunking" OR "split" OR "splitter"
73	- "retriev" OR "search" OR "query"
74	- "langchain" OR "llamaindex" OR "haystack"
75	- "openai.embed" OR "cohere.embed" OR "voyageai"
76	```
77
78	### Step 2: Analyze Each Component
79	For each RAG component found, check against the audit categories above.
80
81	### Step 3: Generate Report
82	Produce a structured audit report:
83
84	```markdown
85	# RAG Audit Report
86
87	## Summary
88	- Files Analyzed: X
89	- Issues Found: Y (X critical, Y warnings, Z suggestions)
90	- Overall Score: X/100
91
92	## Critical Issues
93	[Issues that will cause failures or severe degradation]
94
95	## Warnings
96	[Issues that impact quality or performance]
97
98	## Suggestions
99	[Optimizations and best practices]
100
101	## Detailed Findings
102
103	### [Component Name]
104	Location: `path/to/file.py:line`
105	Issue: [Description]
106	Impact: [What goes wrong]
107	Fix: [How to fix with code example]
108	```
109
110	## Common Anti-Patterns to Flag
111
112	### 1. No Chunk Overlap
113	```python
114	# BAD: No overlap causes context loss at boundaries
115	chunks = text_splitter.split(text, chunk_size=1000, overlap=0)
116
117	# GOOD: 10-20% overlap preserves context
118	chunks = text_splitter.split(text, chunk_size=1000, overlap=150)
119	```
120
121	### 2. Hardcoded Top-K
122	```python
123	# BAD: Fixed top-k regardless of query complexity
124	results = vectorstore.search(query, k=5)
125
126	# GOOD: Dynamic or configurable with score threshold
127	results = vectorstore.search(query, k=10, score_threshold=0.7)
128	```
129
130	### 3. No Reranking
131	```python
132	# BAD: Using raw vector similarity scores only
133	docs = vectorstore.similarity_search(query, k=5)
134	context = "\n".join([d.content for d in docs])
135
136	# GOOD: Rerank for relevance before using
137	docs = vectorstore.similarity_search(query, k=20)
138	reranked = reranker.rerank(query, docs, top_k=5)
139	context = "\n".join([d.content for d in reranked])
140	```
141
142	### 4. Ignoring Metadata
143	```python
144	# BAD: Storing only text
145	vectorstore.add(texts=chunks)
146
147	# GOOD: Preserve source metadata for citations
148	vectorstore.add(
149	texts=chunks,
150	metadatas=[{"source": doc.name, "page": i, "chunk_id": j} for ...]
151	)
152	```
153
154	### 5. No Error Handling
155	```python
156	# BAD: Unhandled failures
157	response = llm.generate(prompt)
158
159	# GOOD: Graceful degradation
160	try:
161	response = llm.generate(prompt)
162	except RateLimitError:
163	response = fallback_response(query)
164	except Exception as e:
165	logger.error(f"Generation failed: {e}")
166	response = "I couldn't process your request. Please try again."
167	```
168
169	### 6. Context Window Overflow
170	```python
171	# BAD: Stuffing all retrieved docs without checking
172	context = "\n".join

People who liked this also viewed

6 picks

Type

azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector searSkillsJul 2026472k1.3k
just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON fromSkillsJul 2026245k37
firecrawlCLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agentsSkillsJul 202694k531
firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "SkillsJul 202676k531
firecrawl-agentAI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, producSkillsJul 202674k531
firecrawl-mapDiscover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where sSkillsJul 202674k531

rag-audit

33stars3forks

View on GitHub

How to install rag-audit?

floflo777/claude-rag-skills/rag-audit

$npx -y skills add floflo777/claude-rag-skills --skill rag-audit

Installs into the current project.

Summary

Analyze RAG (Retrieval-Augmented Generation) implementations for anti-patterns, performance issues, and best practices violations.

Files · 1

View on GitHub

SKILL.md

1	# RAG Audit Skill
2
3	Analyze RAG (Retrieval-Augmented Generation) implementations for anti-patterns, performance issues, and best practices violations.
4
5	## When to Use
6
7	Use `/rag-audit` when:
8	- Reviewing existing RAG code for quality issues
9	- Before deploying a RAG system to production
10	- Debugging retrieval or generation problems
11	- Optimizing RAG pipeline performance
12
13	## What This Skill Does
14
15	1. Code Analysis: Scans your codebase for RAG-related code (embeddings, vector stores, retrieval, generation)
16	2. Anti-Pattern Detection: Identifies common mistakes and suboptimal patterns
17	3. Best Practices Check: Validates against industry standards
18	4. Recommendations: Provides actionable fixes with code examples
19
20	## Audit Categories
21
22	### 1. Chunking Strategy
23	- [ ] Chunk size appropriateness (too large loses precision, too small loses context)
24	- [ ] Overlap configuration (recommended: 10-20% of chunk size)
25	- [ ] Document-type specific chunking (code vs prose vs tables)
26	- [ ] Metadata preservation during chunking
27
28	### 2. Embedding Configuration
29	- [ ] Model selection for use case (multilingual, code, general)
30	- [ ] Dimension efficiency (smaller dims for speed, larger for accuracy)
31	- [ ] Batch processing for large document sets
32	- [ ] Embedding caching to avoid recomputation
33
34	### 3. Vector Store Setup
35	- [ ] Index type selection (HNSW vs IVF vs flat)
36	- [ ] Distance metric matching (cosine for normalized, L2 for raw)
37	- [ ] Collection/namespace organization
38	- [ ] Metadata filtering capabilities
39
40	### 4. Retrieval Pipeline
41	- [ ] Top-k selection (too few misses context, too many adds noise)
42	- [ ] Score thresholding implementation
43	- [ ] Hybrid search (dense + sparse/BM25)
44	- [ ] Reranking stage presence
45	- [ ] Query expansion/transformation
46
47	### 5. Generation Configuration
48	- [ ] Context window utilization
49	- [ ] System prompt quality
50	- [ ] Source citation implementation
51	- [ ] Hallucination guardrails
52	- [ ] Temperature settings for factual tasks
53
54	### 6. Production Readiness
55	- [ ] Error handling and fallbacks
56	- [ ] Logging and observability
57	- [ ] Rate limiting and caching
58	- [ ] Cost optimization (model selection, caching)
59
60	## How to Run an Audit
61
62	When the user invokes `/rag-audit`, follow this process:
63
64	### Step 1: Discover RAG Code
65	Search for RAG-related patterns in the codebase:
66
67	```
68	Patterns to search:
69	- "embedding" OR "embeddings" OR "embed("
70	- "vector" OR "vectorstore" OR "vector_store"
71	- "qdrant" OR "pinecone" OR "chroma" OR "weaviate" OR "milvus"
72	- "chunk" OR "chunking" OR "split" OR "splitter"
73	- "retriev" OR "search" OR "query"
74	- "langchain" OR "llamaindex" OR "haystack"
75	- "openai.embed" OR "cohere.embed" OR "voyageai"
76	```
77
78	### Step 2: Analyze Each Component
79	For each RAG component found, check against the audit categories above.
80
81	### Step 3: Generate Report
82	Produce a structured audit report:
83
84	```markdown
85	# RAG Audit Report
86
87	## Summary
88	- Files Analyzed: X
89	- Issues Found: Y (X critical, Y warnings, Z suggestions)
90	- Overall Score: X/100
91
92	## Critical Issues
93	[Issues that will cause failures or severe degradation]
94
95	## Warnings
96	[Issues that impact quality or performance]
97
98	## Suggestions
99	[Optimizations and best practices]
100
101	## Detailed Findings
102
103	### [Component Name]
104	Location: `path/to/file.py:line`
105	Issue: [Description]
106	Impact: [What goes wrong]
107	Fix: [How to fix with code example]
108	```
109
110	## Common Anti-Patterns to Flag
111
112	### 1. No Chunk Overlap
113	```python
114	# BAD: No overlap causes context loss at boundaries
115	chunks = text_splitter.split(text, chunk_size=1000, overlap=0)
116
117	# GOOD: 10-20% overlap preserves context
118	chunks = text_splitter.split(text, chunk_size=1000, overlap=150)
119	```
120
121	### 2. Hardcoded Top-K
122	```python
123	# BAD: Fixed top-k regardless of query complexity
124	results = vectorstore.search(query, k=5)
125
126	# GOOD: Dynamic or configurable with score threshold
127	results = vectorstore.search(query, k=10, score_threshold=0.7)
128	```
129
130	### 3. No Reranking
131	```python
132	# BAD: Using raw vector similarity scores only
133	docs = vectorstore.similarity_search(query, k=5)
134	context = "\n".join([d.content for d in docs])
135
136	# GOOD: Rerank for relevance before using
137	docs = vectorstore.similarity_search(query, k=20)
138	reranked = reranker.rerank(query, docs, top_k=5)
139	context = "\n".join([d.content for d in reranked])
140	```
141
142	### 4. Ignoring Metadata
143	```python
144	# BAD: Storing only text
145	vectorstore.add(texts=chunks)
146
147	# GOOD: Preserve source metadata for citations
148	vectorstore.add(
149	texts=chunks,
150	metadatas=[{"source": doc.name, "page": i, "chunk_id": j} for ...]
151	)
152	```
153
154	### 5. No Error Handling
155	```python
156	# BAD: Unhandled failures
157	response = llm.generate(prompt)
158
159	# GOOD: Graceful degradation
160	try:
161	response = llm.generate(prompt)
162	except RateLimitError:
163	response = fallback_response(query)
164	except Exception as e:
165	logger.error(f"Generation failed: {e}")
166	response = "I couldn't process your request. Please try again."
167	```
168
169	### 6. Context Window Overflow
170	```python
171	# BAD: Stuffing all retrieved docs without checking
172	context = "\n".join

People who liked this also viewed

6 picks

Type