chunking-advisor

Name: chunking-advisor
Author: floflo777

33stars3forks

How to install chunking-advisor?

floflo777/claude-rag-skills/chunking-advisor

$npx -y skills add floflo777/claude-rag-skills --skill chunking-advisor

Installs into the current project.

Summary

Analyze your documents and recommend optimal chunking strategies based on content type, use case, and embedding model.

Files · 1

View on GitHub

SKILL.md

1	# Chunking Advisor Skill
2
3	Analyze your documents and recommend optimal chunking strategies based on content type, use case, and embedding model.
4
5	## When to Use
6
7	Use `/chunking-advisor` when:
8	- Setting up a new RAG pipeline and unsure about chunk configuration
9	- Experiencing poor retrieval quality (chunks too big or small)
10	- Working with diverse document types (PDFs, code, tables, etc.)
11	- Optimizing an existing chunking strategy
12
13	## Chunking Strategy Decision Tree
14
15	```
16	What type of content?
17	│
18	├── Technical Documentation / Code
19	│ └── Use: Semantic chunking by function/class/section
20	│ Chunk size: 500-1000 tokens
21	│ Overlap: 50-100 tokens
22	│
23	├── Legal / Contracts
24	│ └── Use: Hierarchical chunking (clause → section → document)
25	│ Chunk size: 300-500 tokens
26	│ Overlap: 100-150 tokens (preserve clause boundaries)
27	│
28	├── Product Catalogs
29	│ └── Use: Fixed per-product chunks
30	│ Chunk size: One product = one chunk
31	│ Overlap: None (products are atomic)
32	│
33	├── FAQ / Q&A
34	│ └── Use: Question-answer pairs as chunks
35	│ Chunk size: Variable (complete Q&A)
36	│ Overlap: None
37	│
38	├── Long-form Articles / Blog Posts
39	│ └── Use: Semantic chunking by paragraph/section
40	│ Chunk size: 800-1200 tokens
41	│ Overlap: 100-200 tokens
42	│
43	├── Tables / Structured Data
44	│ └── Use: Row-based or section-based chunking
45	│ Preserve headers in each chunk
46	│ Chunk size: 10-50 rows per chunk
47	│
48	└── Mixed Content
49	└── Use: Document-aware chunking
50	Different strategies per section type
51	Maintain parent-child relationships
52	```
53
54	## Chunking Strategies Explained
55
56	### 1. Fixed-Size Chunking
57	Best for: Homogeneous content, quick implementation
58	Pros: Simple, predictable
59	Cons: May split sentences/paragraphs awkwardly
60
61	```python
62	# Implementation
63	from langchain.text_splitter import CharacterTextSplitter
64
65	splitter = CharacterTextSplitter(
66	chunk_size=1000,
67	chunk_overlap=150,
68	separator="\n\n" # Prefer paragraph breaks
69	)
70	chunks = splitter.split_text(document)
71	```
72
73	Recommended settings by embedding model:
74	\| Model \| Optimal Chunk Size \| Max Chunk Size \|
75	\|-------\|-------------------\|----------------\|
76	\| text-embedding-3-small \| 500-800 tokens \| 8191 tokens \|
77	\| text-embedding-3-large \| 800-1200 tokens \| 8191 tokens \|
78	\| voyage-2 \| 500-1000 tokens \| 4096 tokens \|
79	\| Cohere embed-v3 \| 300-500 tokens \| 512 tokens \|
80
81	### 2. Semantic Chunking
82	Best for: Technical docs, articles, varied content
83	Pros: Respects content boundaries, better retrieval
84	Cons: More complex, requires NLP
85
86	```python
87	# Implementation using sentence boundaries
88	from langchain.text_splitter import RecursiveCharacterTextSplitter
89
90	splitter = RecursiveCharacterTextSplitter(
91	chunk_size=1000,
92	chunk_overlap=150,
93	separators=["\n\n", "\n", ". ", "! ", "? ", ", ", " ", ""]
94	)
95	chunks = splitter.split_text(document)
96	```
97
98	Advanced: Embedding-based semantic chunking
99	```python
100	# Split when semantic similarity drops
101	from sentence_transformers import SentenceTransformer
102	import numpy as np
103
104	def semantic_chunk(sentences, model, threshold=0.5):
105	embeddings = model.encode(sentences)
106	chunks = []
107	current_chunk = [sentences[0]]
108
109	for i in range(1, len(sentences)):
110	similarity = np.dot(embeddings[i], embeddings[i-1])
111	if similarity < threshold:
112	chunks.append(" ".join(current_chunk))
113	current_chunk = []
114	current_chunk.append(sentences[i])
115
116	chunks.append(" ".join(current_chunk))
117	return chunks
118	```
119
120	### 3. Hierarchical Chunking
121	Best for: Legal docs, manuals, structured documents
122	Pros: Preserves document structure, enables multi-level retrieval
123	Cons: Complex implementation
124
125	```python
126	# Parent-child chunk structure
127	class HierarchicalChunker:
128	def chunk(self, document):
129	# Level 1: Sections
130	sections = self.split_by_headers(document)
131
132	chunks = []
133	for section in sections:
134	# Level 2: Paragraphs within sections
135	paragraphs = self.split_paragraphs(section.content)
136
137	for para in paragraphs:
138	chunks.append({
139	"content": para,
140	"metadata": {
141	"parent_section": section.title,
142	"document": document.name,
143	"level": "paragraph"
144	}
145	})
146
147	# Also store section summary for high-level queries
148	chunks.append({
149	"content": section.summary,
150	"metadata": {
151	"document": document.name,
152	"level": "section"
153	}
154	})
155
156	return chunks
157	```
158
159	### 4. Document-Specific Chunking
160	Best for: Mixed document types
161	Pros: Optimal for each content type
162	Cons: Requires content detection
163
164	```python
165	def smart_chunk(document):
166	doc_type = detect_document_type(document)
167
168	if doc_type == "code":
169	return chunk_by_fun

People who liked this also viewed

6 picks

Type

azure-aiUse for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector searSkillsJul 2026472k1.3k
just-scrapeSearch, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON fromSkillsJul 2026245k37
firecrawlCLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agentsSkillsJul 202694k531
firecrawl-searchWeb search with full page content extraction. Use this skill whenever the user asks to search the web, find articles, research a topic, look something up, find recent news, discover sources, or says "SkillsJul 202676k531
firecrawl-agentAI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, producSkillsJul 202674k531
firecrawl-mapDiscover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where sSkillsJul 202674k531

chunking-advisor

33stars3forks

View on GitHub

How to install chunking-advisor?

floflo777/claude-rag-skills/chunking-advisor

$npx -y skills add floflo777/claude-rag-skills --skill chunking-advisor

Installs into the current project.

Summary

Analyze your documents and recommend optimal chunking strategies based on content type, use case, and embedding model.

Files · 1

View on GitHub

SKILL.md

1	# Chunking Advisor Skill
2
3	Analyze your documents and recommend optimal chunking strategies based on content type, use case, and embedding model.
4
5	## When to Use
6
7	Use `/chunking-advisor` when:
8	- Setting up a new RAG pipeline and unsure about chunk configuration
9	- Experiencing poor retrieval quality (chunks too big or small)
10	- Working with diverse document types (PDFs, code, tables, etc.)
11	- Optimizing an existing chunking strategy
12
13	## Chunking Strategy Decision Tree
14
15	```
16	What type of content?
17	│
18	├── Technical Documentation / Code
19	│ └── Use: Semantic chunking by function/class/section
20	│ Chunk size: 500-1000 tokens
21	│ Overlap: 50-100 tokens
22	│
23	├── Legal / Contracts
24	│ └── Use: Hierarchical chunking (clause → section → document)
25	│ Chunk size: 300-500 tokens
26	│ Overlap: 100-150 tokens (preserve clause boundaries)
27	│
28	├── Product Catalogs
29	│ └── Use: Fixed per-product chunks
30	│ Chunk size: One product = one chunk
31	│ Overlap: None (products are atomic)
32	│
33	├── FAQ / Q&A
34	│ └── Use: Question-answer pairs as chunks
35	│ Chunk size: Variable (complete Q&A)
36	│ Overlap: None
37	│
38	├── Long-form Articles / Blog Posts
39	│ └── Use: Semantic chunking by paragraph/section
40	│ Chunk size: 800-1200 tokens
41	│ Overlap: 100-200 tokens
42	│
43	├── Tables / Structured Data
44	│ └── Use: Row-based or section-based chunking
45	│ Preserve headers in each chunk
46	│ Chunk size: 10-50 rows per chunk
47	│
48	└── Mixed Content
49	└── Use: Document-aware chunking
50	Different strategies per section type
51	Maintain parent-child relationships
52	```
53
54	## Chunking Strategies Explained
55
56	### 1. Fixed-Size Chunking
57	Best for: Homogeneous content, quick implementation
58	Pros: Simple, predictable
59	Cons: May split sentences/paragraphs awkwardly
60
61	```python
62	# Implementation
63	from langchain.text_splitter import CharacterTextSplitter
64
65	splitter = CharacterTextSplitter(
66	chunk_size=1000,
67	chunk_overlap=150,
68	separator="\n\n" # Prefer paragraph breaks
69	)
70	chunks = splitter.split_text(document)
71	```
72
73	Recommended settings by embedding model:
74	\| Model \| Optimal Chunk Size \| Max Chunk Size \|
75	\|-------\|-------------------\|----------------\|
76	\| text-embedding-3-small \| 500-800 tokens \| 8191 tokens \|
77	\| text-embedding-3-large \| 800-1200 tokens \| 8191 tokens \|
78	\| voyage-2 \| 500-1000 tokens \| 4096 tokens \|
79	\| Cohere embed-v3 \| 300-500 tokens \| 512 tokens \|
80
81	### 2. Semantic Chunking
82	Best for: Technical docs, articles, varied content
83	Pros: Respects content boundaries, better retrieval
84	Cons: More complex, requires NLP
85
86	```python
87	# Implementation using sentence boundaries
88	from langchain.text_splitter import RecursiveCharacterTextSplitter
89
90	splitter = RecursiveCharacterTextSplitter(
91	chunk_size=1000,
92	chunk_overlap=150,
93	separators=["\n\n", "\n", ". ", "! ", "? ", ", ", " ", ""]
94	)
95	chunks = splitter.split_text(document)
96	```
97
98	Advanced: Embedding-based semantic chunking
99	```python
100	# Split when semantic similarity drops
101	from sentence_transformers import SentenceTransformer
102	import numpy as np
103
104	def semantic_chunk(sentences, model, threshold=0.5):
105	embeddings = model.encode(sentences)
106	chunks = []
107	current_chunk = [sentences[0]]
108
109	for i in range(1, len(sentences)):
110	similarity = np.dot(embeddings[i], embeddings[i-1])
111	if similarity < threshold:
112	chunks.append(" ".join(current_chunk))
113	current_chunk = []
114	current_chunk.append(sentences[i])
115
116	chunks.append(" ".join(current_chunk))
117	return chunks
118	```
119
120	### 3. Hierarchical Chunking
121	Best for: Legal docs, manuals, structured documents
122	Pros: Preserves document structure, enables multi-level retrieval
123	Cons: Complex implementation
124
125	```python
126	# Parent-child chunk structure
127	class HierarchicalChunker:
128	def chunk(self, document):
129	# Level 1: Sections
130	sections = self.split_by_headers(document)
131
132	chunks = []
133	for section in sections:
134	# Level 2: Paragraphs within sections
135	paragraphs = self.split_paragraphs(section.content)
136
137	for para in paragraphs:
138	chunks.append({
139	"content": para,
140	"metadata": {
141	"parent_section": section.title,
142	"document": document.name,
143	"level": "paragraph"
144	}
145	})
146
147	# Also store section summary for high-level queries
148	chunks.append({
149	"content": section.summary,
150	"metadata": {
151	"document": document.name,
152	"level": "section"
153	}
154	})
155
156	return chunks
157	```
158
159	### 4. Document-Specific Chunking
160	Best for: Mixed document types
161	Pros: Optimal for each content type
162	Cons: Requires content detection
163
164	```python
165	def smart_chunk(document):
166	doc_type = detect_document_type(document)
167
168	if doc_type == "code":
169	return chunk_by_fun

People who liked this also viewed

6 picks

Type