building-rag-systems

Name: building-rag-systems
Author: mjunaidca

20installs11forks

How to install building-rag-systems?

mjunaidca/mjs-agent-skills/building-rag-systems

$npx -y skills add mjunaidca/mjs-agent-skills --skill building-rag-systems

Installs into the current project.

Summary

Build production RAG systems with semantic chunking, incremental indexing, and filtered retrieval. Use when implementing document ingestion pipelines, vector search with Qdrant, or context-aware retrieval. Covers chunking strategies, change detection, payload indexing, and contex

Files · 1

View on GitHub

SKILL.md

1	# Building RAG Systems
2
3	Production-grade RAG with semantic chunking, incremental updates, and filtered retrieval.
4
5	## Quick Start
6
7	```bash
8	# Dependencies
9	pip install qdrant-client openai pydantic python-frontmatter
10
11	# Core components
12	# 1. Crawler → discovers files, extracts path metadata
13	# 2. Parser → extracts frontmatter, computes file hash
14	# 3. Chunker → semantic split on ## headers, 400 tokens, 15% overlap
15	# 4. Embedder → batched OpenAI embeddings
16	# 5. Uploader → Qdrant upsert with indexed payloads
17	```
18
19	---
20
21	## Ingestion Pipeline
22
23	### Architecture
24
25	```
26	┌──────────┐ ┌────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐
27	│ Crawler │ -> │ Parser │ -> │ Chunker │ -> │ Embedder │ -> │ Uploader │
28	└──────────┘ └────────┘ └─────────┘ └──────────┘ └──────────┘
29	│ │ │ │ │
30	Discovers Extracts Splits by Generates Upserts to
31	files frontmatter semantic vectors Qdrant
32	+ file hash boundaries (batched) (batched)
33	```
34
35	### Semantic Chunking (NOT Fixed-Size)
36
37	```python
38	class SemanticChunker:
39	"""
40	Production chunking:
41	- Split on ## headers (semantic boundaries)
42	- Target 400 tokens (NVIDIA benchmark optimal)
43	- 15% overlap for context continuity
44	- Track prev/next for context expansion
45	"""
46	SECTION_PATTERN = re.compile(r"(?=^## )", re.MULTILINE)
47	TOKENS_PER_WORD = 1.3
48
49	def __init__(
50	self,
51	target_tokens: int = 400,
52	max_tokens: int = 512,
53	overlap_percent: float = 0.15,
54	):
55	self.target_words = int(target_tokens / self.TOKENS_PER_WORD)
56	self.overlap_words = int(self.target_words * overlap_percent)
57
58	def chunk(self, content: str, file_hash: str) -> list[Chunk]:
59	sections = self.SECTION_PATTERN.split(content)
60	chunks = []
61
62	for idx, section in enumerate(sections):
63	content_hash = hashlib.sha256(section.encode()).hexdigest()[:16]
64	chunk_id = f"{file_hash[:8]}_{content_hash}_{idx}"
65
66	chunks.append(Chunk(
67	id=chunk_id,
68	text=section,
69	chunk_index=idx,
70	total_chunks=len(sections),
71	prev_chunk_id=chunks[-1].id if chunks else None,
72	content_hash=content_hash,
73	source_file_hash=file_hash,
74	))
75
76	# Set next_chunk_id on previous
77	if len(chunks) > 1:
78	chunks[-2].next_chunk_id = chunk_id
79
80	return chunks
81	```
82
83	### Change Detection (Incremental Updates)
84
85	```python
86	def compute_file_hash(file_path: str) -> str:
87	"""SHA-256 for change detection."""
88	with open(file_path, 'rb') as f:
89	return hashlib.sha256(f.read()).hexdigest()
90
91	class QdrantStateTracker:
92	"""Query Qdrant payloads directly - no external state DB needed."""
93
94	def get_indexed_files(self, book_id: str) -> dict[str, str]:
95	"""Returns {file_path: file_hash} from Qdrant."""
96	indexed = {}
97	offset = None
98
99	while True:
100	points, next_offset = self.client.scroll(
101	collection_name=self.collection,
102	scroll_filter=Filter(must=[
103	FieldCondition(key="book_id", match=MatchValue(value=book_id))
104	]),
105	limit=100,
106	offset=offset,
107	with_payload=["source_file", "source_file_hash"],
108	with_vectors=False,
109	)
110
111	for point in points:
112	indexed[point.payload["source_file"]] = point.payload["source_file_hash"]
113
114	if next_offset is None:
115	break
116	offset = next_offset
117
118	return indexed
119
120	def detect_changes(self, current: dict[str, str], indexed: dict[str, str]):
121	"""Compare filesystem vs index."""
122	new = [p for p in current if p not in indexed]
123	deleted = [p for p in indexed if p not in current]
124	modified = [p for p in current if p in indexed and current[p] != indexed[p]]
125	return new, modified, deleted
126	```
127
128	### Batched Embeddings
129
130	```python
131	class OpenAIEmbedder:
132	def __init__(self, model: str = "text-embedding-3-small", batch_size: int = 20):
133	self.client = OpenAI()
134	self.model = model
135	self.batch_size = batch_size # OpenAI recommendation
136
137	def embed_chunks(self, chunks: list[Chunk]) -> list[EmbeddedChunk]:
138	embedded = []
139	for i in range(0, len(chunks), self.batch_size):
140	batch = chunks[i:i + self.batch_size]

People who liked this also viewed

6 picks

Type

building-rag-systems

20installs11forks

View on GitHub

How to install building-rag-systems?

mjunaidca/mjs-agent-skills/building-rag-systems

$npx -y skills add mjunaidca/mjs-agent-skills --skill building-rag-systems

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Building RAG Systems
2
3	Production-grade RAG with semantic chunking, incremental updates, and filtered retrieval.
4
5	## Quick Start
6
7	```bash
8	# Dependencies
9	pip install qdrant-client openai pydantic python-frontmatter
10
11	# Core components
12	# 1. Crawler → discovers files, extracts path metadata
13	# 2. Parser → extracts frontmatter, computes file hash
14	# 3. Chunker → semantic split on ## headers, 400 tokens, 15% overlap
15	# 4. Embedder → batched OpenAI embeddings
16	# 5. Uploader → Qdrant upsert with indexed payloads
17	```
18
19	---
20
21	## Ingestion Pipeline
22
23	### Architecture
24
25	```
26	┌──────────┐ ┌────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐
27	│ Crawler │ -> │ Parser │ -> │ Chunker │ -> │ Embedder │ -> │ Uploader │
28	└──────────┘ └────────┘ └─────────┘ └──────────┘ └──────────┘
29	│ │ │ │ │
30	Discovers Extracts Splits by Generates Upserts to
31	files frontmatter semantic vectors Qdrant
32	+ file hash boundaries (batched) (batched)
33	```
34
35	### Semantic Chunking (NOT Fixed-Size)
36
37	```python
38	class SemanticChunker:
39	"""
40	Production chunking:
41	- Split on ## headers (semantic boundaries)
42	- Target 400 tokens (NVIDIA benchmark optimal)
43	- 15% overlap for context continuity
44	- Track prev/next for context expansion
45	"""
46	SECTION_PATTERN = re.compile(r"(?=^## )", re.MULTILINE)
47	TOKENS_PER_WORD = 1.3
48
49	def __init__(
50	self,
51	target_tokens: int = 400,
52	max_tokens: int = 512,
53	overlap_percent: float = 0.15,
54	):
55	self.target_words = int(target_tokens / self.TOKENS_PER_WORD)
56	self.overlap_words = int(self.target_words * overlap_percent)
57
58	def chunk(self, content: str, file_hash: str) -> list[Chunk]:
59	sections = self.SECTION_PATTERN.split(content)
60	chunks = []
61
62	for idx, section in enumerate(sections):
63	content_hash = hashlib.sha256(section.encode()).hexdigest()[:16]
64	chunk_id = f"{file_hash[:8]}_{content_hash}_{idx}"
65
66	chunks.append(Chunk(
67	id=chunk_id,
68	text=section,
69	chunk_index=idx,
70	total_chunks=len(sections),
71	prev_chunk_id=chunks[-1].id if chunks else None,
72	content_hash=content_hash,
73	source_file_hash=file_hash,
74	))
75
76	# Set next_chunk_id on previous
77	if len(chunks) > 1:
78	chunks[-2].next_chunk_id = chunk_id
79
80	return chunks
81	```
82
83	### Change Detection (Incremental Updates)
84
85	```python
86	def compute_file_hash(file_path: str) -> str:
87	"""SHA-256 for change detection."""
88	with open(file_path, 'rb') as f:
89	return hashlib.sha256(f.read()).hexdigest()
90
91	class QdrantStateTracker:
92	"""Query Qdrant payloads directly - no external state DB needed."""
93
94	def get_indexed_files(self, book_id: str) -> dict[str, str]:
95	"""Returns {file_path: file_hash} from Qdrant."""
96	indexed = {}
97	offset = None
98
99	while True:
100	points, next_offset = self.client.scroll(
101	collection_name=self.collection,
102	scroll_filter=Filter(must=[
103	FieldCondition(key="book_id", match=MatchValue(value=book_id))
104	]),
105	limit=100,
106	offset=offset,
107	with_payload=["source_file", "source_file_hash"],
108	with_vectors=False,
109	)
110
111	for point in points:
112	indexed[point.payload["source_file"]] = point.payload["source_file_hash"]
113
114	if next_offset is None:
115	break
116	offset = next_offset
117
118	return indexed
119
120	def detect_changes(self, current: dict[str, str], indexed: dict[str, str]):
121	"""Compare filesystem vs index."""
122	new = [p for p in current if p not in indexed]
123	deleted = [p for p in indexed if p not in current]
124	modified = [p for p in current if p in indexed and current[p] != indexed[p]]
125	return new, modified, deleted
126	```
127
128	### Batched Embeddings
129
130	```python
131	class OpenAIEmbedder:
132	def __init__(self, model: str = "text-embedding-3-small", batch_size: int = 20):
133	self.client = OpenAI()
134	self.model = model
135	self.batch_size = batch_size # OpenAI recommendation
136
137	def embed_chunks(self, chunks: list[Chunk]) -> list[EmbeddedChunk]:
138	embedded = []
139	for i in range(0, len(chunks), self.batch_size):
140	batch = chunks[i:i + self.batch_size]

People who liked this also viewed

6 picks

Type