chunking-strategy

Name: chunking-strategy
Author: giuseppe-trisciuoglio

1.7kinstalls36forks

How to install chunking-strategy?

giuseppe-trisciuoglio/developer-kit/chunking-strategy

$npx -y skills add giuseppe-trisciuoglio/developer-kit --skill chunking-strategy

Installs into the current project.

Summary

Provides chunking strategies for RAG systems. Generates chunk size recommendations (256-1024 tokens), overlap percentages (10-20%), and semantic boundary detection methods. Validates semantic coherence and evaluates retrieval precision/recall metrics. Use when building retrieval-

Files · 1

View on GitHub

SKILL.md

1	# Chunking Strategy for RAG Systems
2
3	## Overview
4
5	Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.
6
7	## When to Use
8
9	Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.
10
11	## Instructions
12
13	### Choose Chunking Strategy
14
15	Select based on document type and use case:
16
17	1. Fixed-Size Chunking (Level 1)
18	- Use for simple documents without clear structure
19	- Start with 512 tokens and 10-20% overlap
20	- Adjust: 256 for factoid queries, 1024 for analytical
21
22	2. Recursive Character Chunking (Level 2)
23	- Use for documents with structural boundaries
24	- Hierarchical separators: paragraphs → sentences → words
25	- Customize for document types (HTML, Markdown, JSON)
26
27	3. Structure-Aware Chunking (Level 3)
28	- Use for structured content (Markdown, code, tables, PDFs)
29	- Preserve semantic units: functions, sections, table blocks
30	- Validate structure preservation post-split
31
32	4. Semantic Chunking (Level 4)
33	- Use for complex documents with thematic shifts
34	- Embedding-based boundary detection with 0.8 similarity threshold
35	- Buffer size: 3-5 sentences
36
37	5. Advanced Methods (Level 5)
38	- Late Chunking for long-context models
39	- Contextual Retrieval for high-precision requirements
40	- Monitor computational cost vs. retrieval gain
41
42	Reference: [references/strategies.md](references/strategies.md).
43
44	### Implement Chunking Pipeline
45
46	1. Pre-process documents
47	- Analyze structure, content types, information density
48	- Identify multi-modal content (tables, images, code)
49
50	2. Select parameters
51	- Chunk size: embedding model context window / 4
52	- Overlap: 10-20% for most cases
53	- Strategy-specific settings
54
55	3. Process and validate
56	- Apply chunking strategy
57	- Validate coherence: run `evaluate_chunks.py --coherence` (see below)
58	- Test with representative documents
59
60	4. Evaluate and iterate
61	- Measure precision and recall
62	- If precision < 0.7: reduce chunk_size by 25% and re-evaluate
63	- If recall < 0.6: increase overlap by 10% and re-evaluate
64	- Monitor latency and memory usage
65
66	Reference: [references/implementation.md](references/implementation.md).
67
68	### Validate Chunk Quality
69
70	Run validation commands to assess chunk quality:
71
72	```bash
73	# Check semantic coherence (requires sentence-transformers)
74	python -c "
75	from sentence_transformers import SentenceTransformer
76	model = SentenceTransformer('all-MiniLM-L6-v2')
77	chunks = [...] # your chunks
78	embeddings = model.encode(chunks)
79	similarity = (embeddings @ embeddings.T).mean()
80	print(f'Cohesion: {similarity:.3f}') # target: 0.3-0.7
81	"
82
83	# Measure retrieval precision
84	python -c "
85	relevant = sum(1 for c in retrieved if c in relevant_chunks)
86	precision = relevant / len(retrieved)
87	print(f'Precision: {precision:.2f}') # target: >= 0.7
88	"
89
90	# Check chunk size distribution
91	python -c "
92	import numpy as np
93	sizes = [len(c.split()) for c in chunks]
94	print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
95	print(f'Min: {min(sizes)}, Max: {max(sizes)}')
96	"
97	```
98
99	Reference: [references/evaluation.md](references/evaluation.md).
100
101	## Examples
102
103	### Fixed-Size Chunking
104
105	```python
106	from langchain.text_splitter import RecursiveCharacterTextSplitter
107
108	splitter = RecursiveCharacterTextSplitter(
109	chunk_size=256,
110	chunk_overlap=25,
111	length_function=len
112	)
113	chunks = splitter.split_documents(documents)
114	```
115
116	### Structure-Aware Code Chunking
117
118	```python
119	import ast
120
121	def chunk_python_code(code):
122	tree = ast.parse(code)
123	chunks = []
124	for node in ast.walk(tree):
125	if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
126	chunks.append(ast.get_source_segment(code, node))
127	return chunks
128	```
129
130	### Semantic Chunking
131
132	```python
133	def semantic_chunk(text, similarity_threshold=0.8):
134	sentences = split_into_sentences(text)
135	embeddings = generate_embeddings(sentences)
136	chunks, current = [], [sentences[0]]
137	for i in range(1, len(sentences)):
138	sim = cosine_similarity(embeddings[i-1], embeddings[i])
139	if sim < similarity_threshold:
140	chunks.append(" ".join(current))
141	current = [sentences[i]]
142	else:
143	current.append(sentences[i])
144	chunks.append(" ".join(current))
145	return chunks
146	```
147
148	## Best Practices
149
150	### Core Principles
151	- Balance context preservation with retri

People who liked this also viewed

6 picks

Type

chunking-strategy

1.7kinstalls36forks

View on GitHub

How to install chunking-strategy?

giuseppe-trisciuoglio/developer-kit/chunking-strategy

$npx -y skills add giuseppe-trisciuoglio/developer-kit --skill chunking-strategy

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Chunking Strategy for RAG Systems
2
3	## Overview
4
5	Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.
6
7	## When to Use
8
9	Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.
10
11	## Instructions
12
13	### Choose Chunking Strategy
14
15	Select based on document type and use case:
16
17	1. Fixed-Size Chunking (Level 1)
18	- Use for simple documents without clear structure
19	- Start with 512 tokens and 10-20% overlap
20	- Adjust: 256 for factoid queries, 1024 for analytical
21
22	2. Recursive Character Chunking (Level 2)
23	- Use for documents with structural boundaries
24	- Hierarchical separators: paragraphs → sentences → words
25	- Customize for document types (HTML, Markdown, JSON)
26
27	3. Structure-Aware Chunking (Level 3)
28	- Use for structured content (Markdown, code, tables, PDFs)
29	- Preserve semantic units: functions, sections, table blocks
30	- Validate structure preservation post-split
31
32	4. Semantic Chunking (Level 4)
33	- Use for complex documents with thematic shifts
34	- Embedding-based boundary detection with 0.8 similarity threshold
35	- Buffer size: 3-5 sentences
36
37	5. Advanced Methods (Level 5)
38	- Late Chunking for long-context models
39	- Contextual Retrieval for high-precision requirements
40	- Monitor computational cost vs. retrieval gain
41
42	Reference: [references/strategies.md](references/strategies.md).
43
44	### Implement Chunking Pipeline
45
46	1. Pre-process documents
47	- Analyze structure, content types, information density
48	- Identify multi-modal content (tables, images, code)
49
50	2. Select parameters
51	- Chunk size: embedding model context window / 4
52	- Overlap: 10-20% for most cases
53	- Strategy-specific settings
54
55	3. Process and validate
56	- Apply chunking strategy
57	- Validate coherence: run `evaluate_chunks.py --coherence` (see below)
58	- Test with representative documents
59
60	4. Evaluate and iterate
61	- Measure precision and recall
62	- If precision < 0.7: reduce chunk_size by 25% and re-evaluate
63	- If recall < 0.6: increase overlap by 10% and re-evaluate
64	- Monitor latency and memory usage
65
66	Reference: [references/implementation.md](references/implementation.md).
67
68	### Validate Chunk Quality
69
70	Run validation commands to assess chunk quality:
71
72	```bash
73	# Check semantic coherence (requires sentence-transformers)
74	python -c "
75	from sentence_transformers import SentenceTransformer
76	model = SentenceTransformer('all-MiniLM-L6-v2')
77	chunks = [...] # your chunks
78	embeddings = model.encode(chunks)
79	similarity = (embeddings @ embeddings.T).mean()
80	print(f'Cohesion: {similarity:.3f}') # target: 0.3-0.7
81	"
82
83	# Measure retrieval precision
84	python -c "
85	relevant = sum(1 for c in retrieved if c in relevant_chunks)
86	precision = relevant / len(retrieved)
87	print(f'Precision: {precision:.2f}') # target: >= 0.7
88	"
89
90	# Check chunk size distribution
91	python -c "
92	import numpy as np
93	sizes = [len(c.split()) for c in chunks]
94	print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
95	print(f'Min: {min(sizes)}, Max: {max(sizes)}')
96	"
97	```
98
99	Reference: [references/evaluation.md](references/evaluation.md).
100
101	## Examples
102
103	### Fixed-Size Chunking
104
105	```python
106	from langchain.text_splitter import RecursiveCharacterTextSplitter
107
108	splitter = RecursiveCharacterTextSplitter(
109	chunk_size=256,
110	chunk_overlap=25,
111	length_function=len
112	)
113	chunks = splitter.split_documents(documents)
114	```
115
116	### Structure-Aware Code Chunking
117
118	```python
119	import ast
120
121	def chunk_python_code(code):
122	tree = ast.parse(code)
123	chunks = []
124	for node in ast.walk(tree):
125	if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
126	chunks.append(ast.get_source_segment(code, node))
127	return chunks
128	```
129
130	### Semantic Chunking
131
132	```python
133	def semantic_chunk(text, similarity_threshold=0.8):
134	sentences = split_into_sentences(text)
135	embeddings = generate_embeddings(sentences)
136	chunks, current = [], [sentences[0]]
137	for i in range(1, len(sentences)):
138	sim = cosine_similarity(embeddings[i-1], embeddings[i])
139	if sim < similarity_threshold:
140	chunks.append(" ".join(current))
141	current = [sentences[i]]
142	else:
143	current.append(sentences[i])
144	chunks.append(" ".join(current))
145	return chunks
146	```
147
148	## Best Practices
149
150	### Core Principles
151	- Balance context preservation with retri

People who liked this also viewed

6 picks

Type