ai-data-engineering

Name: ai-data-engineering
Author: ancoleman

49installs59forks

How to install ai-data-engineering?

ancoleman/ai-design-components/ai-data-engineering

$npx -y skills add ancoleman/ai-design-components --skill ai-data-engineering

Installs into the current project.

Summary

Data pipelines, feature stores, and embedding generation for AI/ML systems. Use when building RAG pipelines, ML feature serving, or data transformations. Covers feature stores (Feast, Tecton), embedding pipelines, chunking strategies, orchestration (Dagster, Prefect, Airflow), db

Files · 1

View on GitHub

SKILL.md

1	# AI Data Engineering
2
3	## Purpose
4
5	Build data infrastructure for AI/ML systems including RAG pipelines, feature stores, and embedding generation. Provides architecture patterns, orchestration workflows, and evaluation metrics for production AI applications.
6
7	## When to Use
8
9	Use this skill when:
10	- Building RAG (Retrieval-Augmented Generation) pipelines
11	- Implementing semantic search or vector databases
12	- Setting up ML feature stores for real-time serving
13	- Creating embedding generation pipelines
14	- Evaluating RAG quality with RAGAS metrics
15	- Orchestrating data workflows for AI systems
16	- Integrating with frontend skills (ai-chat, search-filter)
17
18	Skip this skill if:
19	- Building traditional CRUD applications (use databases-relational)
20	- Simple key-value storage (use databases-nosql)
21	- No AI/ML components in the application
22
23	## RAG Pipeline Architecture
24
25	RAG pipelines have 5 distinct stages. Understanding this architecture is critical for production implementations.
26
27	```
28	┌─────────────────────────────────────────────────────────────┐
29	│ RAG Pipeline (5 Stages) │
30	├─────────────────────────────────────────────────────────────┤
31	│ │
32	│ 1. INGESTION → Load documents (PDF, DOCX, Markdown) │
33	│ 2. INDEXING → Chunk (512 tokens) + Embed + Store │
34	│ 3. RETRIEVAL → Query embedding + Vector search + Filters │
35	│ 4. GENERATION → Context injection + LLM streaming │
36	│ 5. EVALUATION → RAGAS metrics (faithfulness, relevancy) │
37	│ │
38	└─────────────────────────────────────────────────────────────┘
39	```
40
41	For complete RAG architecture with implementation patterns, see:
42	- `references/rag-architecture.md` - Detailed 5-stage breakdown
43	- `examples/langchain-rag/basic_rag.py` - Working implementation
44
45	## Chunking Strategies
46
47	Chunking is the most critical decision for RAG quality. Poor chunking breaks retrieval.
48
49	Default Recommendation:
50	- Size: 512 tokens
51	- Overlap: 50-100 tokens
52	- Method: Fixed token-based
53
54	Why these values:
55	- Too small (<256 tokens): Loses context, requires many retrievals
56	- Too large (>1024 tokens): Includes irrelevant content, hits token limits
57	- Overlap prevents information loss at chunk boundaries
58
59	Alternative strategies for special cases:
60
61	```python
62	# Code-aware chunking (preserves functions/classes)
63	from langchain.text_splitter import RecursiveCharacterTextSplitter
64
65	code_splitter = RecursiveCharacterTextSplitter.from_language(
66	language="python",
67	chunk_size=512,
68	chunk_overlap=50
69	)
70
71	# Semantic chunking (splits on meaning, not tokens)
72	from langchain.text_splitter import SemanticChunker
73
74	semantic_splitter = SemanticChunker(
75	embeddings=embeddings,
76	breakpoint_threshold_type="percentile" # Split at semantic boundaries
77	)
78	```
79
80	See: `references/chunking-strategies.md` for complete decision framework
81
82	## Embedding Generation
83
84	Embedding quality directly impacts retrieval accuracy. Voyage AI is currently best-in-class.
85
86	Primary Recommendation: Voyage AI voyage-3
87	- Dimensions: 1024
88	- MTEB Score: 69.0 (highest as of Dec 2025)
89	- Cost: $$$ but 9.74% better than OpenAI
90	- Use for: Production systems requiring best retrieval quality
91
92	Cost-Effective Alternative: OpenAI text-embedding-3-small
93	- Dimensions: 1536
94	- MTEB Score: 62.3
95	- Cost: $ (5x cheaper than voyage-3)
96	- Use for: Development, prototyping, cost-sensitive applications
97
98	Implementation:
99
100	```python
101	from langchain_voyageai import VoyageAIEmbeddings
102	from langchain_openai import OpenAIEmbeddings
103
104	# Production (best quality)
105	embeddings = VoyageAIEmbeddings(
106	model="voyage-3",
107	voyage_api_key="your-api-key"
108	)
109
110	# Development (cost-effective)
111	embeddings = OpenAIEmbeddings(
112	model="text-embedding-3-small",
113	openai_api_key="your-api-key"
114	)
115	```
116
117	See: `references/embedding-strategies.md` for complete provider comparison
118
119	## RAGAS Evaluation Metrics
120
121	Traditional metrics (BLEU, ROUGE) don't measure RAG quality. RAGAS provides LLM-as-judge evaluation.
122
123	4 Core Metrics:
124
125	\| Metric \| Measures \| Good Score \|
126	\|--------\|----------\|------------\|
127	\| Faithfulness \| Factual consistency with retrieved context \| > 0.8 \|
128	\| Answer Relevancy \| Does answer address the user's question? \| > 0.7 \|
129	\| Context Precision \| Are retrieved chunks actually relevant? \| > 0.6 \|
130	\| Context Recall \| Were all necessary chunks retrieved? \| > 0.7 \|
131
132	Quick evaluation script:
133
134	```bash
135	# Run RAGAS evaluation (TOKEN-FREE script execu

People who liked this also viewed

6 picks

Type

ai-data-engineering

49installs59forks

View on GitHub

How to install ai-data-engineering?

ancoleman/ai-design-components/ai-data-engineering

$npx -y skills add ancoleman/ai-design-components --skill ai-data-engineering

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AI Data Engineering
2
3	## Purpose
4
5	Build data infrastructure for AI/ML systems including RAG pipelines, feature stores, and embedding generation. Provides architecture patterns, orchestration workflows, and evaluation metrics for production AI applications.
6
7	## When to Use
8
9	Use this skill when:
10	- Building RAG (Retrieval-Augmented Generation) pipelines
11	- Implementing semantic search or vector databases
12	- Setting up ML feature stores for real-time serving
13	- Creating embedding generation pipelines
14	- Evaluating RAG quality with RAGAS metrics
15	- Orchestrating data workflows for AI systems
16	- Integrating with frontend skills (ai-chat, search-filter)
17
18	Skip this skill if:
19	- Building traditional CRUD applications (use databases-relational)
20	- Simple key-value storage (use databases-nosql)
21	- No AI/ML components in the application
22
23	## RAG Pipeline Architecture
24
25	RAG pipelines have 5 distinct stages. Understanding this architecture is critical for production implementations.
26
27	```
28	┌─────────────────────────────────────────────────────────────┐
29	│ RAG Pipeline (5 Stages) │
30	├─────────────────────────────────────────────────────────────┤
31	│ │
32	│ 1. INGESTION → Load documents (PDF, DOCX, Markdown) │
33	│ 2. INDEXING → Chunk (512 tokens) + Embed + Store │
34	│ 3. RETRIEVAL → Query embedding + Vector search + Filters │
35	│ 4. GENERATION → Context injection + LLM streaming │
36	│ 5. EVALUATION → RAGAS metrics (faithfulness, relevancy) │
37	│ │
38	└─────────────────────────────────────────────────────────────┘
39	```
40
41	For complete RAG architecture with implementation patterns, see:
42	- `references/rag-architecture.md` - Detailed 5-stage breakdown
43	- `examples/langchain-rag/basic_rag.py` - Working implementation
44
45	## Chunking Strategies
46
47	Chunking is the most critical decision for RAG quality. Poor chunking breaks retrieval.
48
49	Default Recommendation:
50	- Size: 512 tokens
51	- Overlap: 50-100 tokens
52	- Method: Fixed token-based
53
54	Why these values:
55	- Too small (<256 tokens): Loses context, requires many retrievals
56	- Too large (>1024 tokens): Includes irrelevant content, hits token limits
57	- Overlap prevents information loss at chunk boundaries
58
59	Alternative strategies for special cases:
60
61	```python
62	# Code-aware chunking (preserves functions/classes)
63	from langchain.text_splitter import RecursiveCharacterTextSplitter
64
65	code_splitter = RecursiveCharacterTextSplitter.from_language(
66	language="python",
67	chunk_size=512,
68	chunk_overlap=50
69	)
70
71	# Semantic chunking (splits on meaning, not tokens)
72	from langchain.text_splitter import SemanticChunker
73
74	semantic_splitter = SemanticChunker(
75	embeddings=embeddings,
76	breakpoint_threshold_type="percentile" # Split at semantic boundaries
77	)
78	```
79
80	See: `references/chunking-strategies.md` for complete decision framework
81
82	## Embedding Generation
83
84	Embedding quality directly impacts retrieval accuracy. Voyage AI is currently best-in-class.
85
86	Primary Recommendation: Voyage AI voyage-3
87	- Dimensions: 1024
88	- MTEB Score: 69.0 (highest as of Dec 2025)
89	- Cost: $$$ but 9.74% better than OpenAI
90	- Use for: Production systems requiring best retrieval quality
91
92	Cost-Effective Alternative: OpenAI text-embedding-3-small
93	- Dimensions: 1536
94	- MTEB Score: 62.3
95	- Cost: $ (5x cheaper than voyage-3)
96	- Use for: Development, prototyping, cost-sensitive applications
97
98	Implementation:
99
100	```python
101	from langchain_voyageai import VoyageAIEmbeddings
102	from langchain_openai import OpenAIEmbeddings
103
104	# Production (best quality)
105	embeddings = VoyageAIEmbeddings(
106	model="voyage-3",
107	voyage_api_key="your-api-key"
108	)
109
110	# Development (cost-effective)
111	embeddings = OpenAIEmbeddings(
112	model="text-embedding-3-small",
113	openai_api_key="your-api-key"
114	)
115	```
116
117	See: `references/embedding-strategies.md` for complete provider comparison
118
119	## RAGAS Evaluation Metrics
120
121	Traditional metrics (BLEU, ROUGE) don't measure RAG quality. RAGAS provides LLM-as-judge evaluation.
122
123	4 Core Metrics:
124
125	\| Metric \| Measures \| Good Score \|
126	\|--------\|----------\|------------\|
127	\| Faithfulness \| Factual consistency with retrieved context \| > 0.8 \|
128	\| Answer Relevancy \| Does answer address the user's question? \| > 0.7 \|
129	\| Context Precision \| Are retrieved chunks actually relevant? \| > 0.6 \|
130	\| Context Recall \| Were all necessary chunks retrieved? \| > 0.7 \|
131
132	Quick evaluation script:
133
134	```bash
135	# Run RAGAS evaluation (TOKEN-FREE script execu

People who liked this also viewed

6 picks

Type