embedding-optimization

Name: embedding-optimization
Author: ancoleman

55installs59forks

How to install embedding-optimization?

ancoleman/ai-design-components/embedding-optimization

$npx -y skills add ancoleman/ai-design-components --skill embedding-optimization

Installs into the current project.

Summary

Optimizing vector embeddings for RAG systems through model selection, chunking strategies, caching, and performance tuning. Use when building semantic search, RAG pipelines, or document retrieval systems that require cost-effective, high-quality embeddings.

Files · 1

View on GitHub

SKILL.md

1	# Embedding Optimization
2
3	Optimize embedding generation for cost, performance, and quality in RAG and semantic search systems.
4
5	## When to Use This Skill
6
7	Trigger this skill when:
8	- Building RAG (Retrieval Augmented Generation) systems
9	- Implementing semantic search or similarity detection
10	- Optimizing embedding API costs (reducing by 70-90%)
11	- Improving document retrieval quality through better chunking
12	- Processing large document corpora (thousands to millions of documents)
13	- Selecting between API-based vs. local embedding models
14
15	## Model Selection Framework
16
17	Choose the optimal embedding model based on requirements:
18
19	Quick Recommendations:
20	- Startup/MVP: `all-MiniLM-L6-v2` (local, 384 dims, zero API costs)
21	- Production: `text-embedding-3-small` (API, 1,536 dims, balanced quality/cost)
22	- High Quality: `text-embedding-3-large` (API, 3,072 dims, premium)
23	- Multilingual: `multilingual-e5-base` (local, 768 dims) or Cohere `embed-multilingual-v3.0`
24
25	For detailed decision frameworks including cost comparisons, quality benchmarks, and data privacy considerations, see `references/model-selection-guide.md`.
26
27	Model Comparison Summary:
28
29	\| Model \| Type \| Dimensions \| Cost per 1M tokens \| Best For \|
30	\|-------\|------\|-----------\|-------------------\|----------\|
31	\| all-MiniLM-L6-v2 \| Local \| 384 \| $0 (compute only) \| High volume, tight budgets \|
32	\| BGE-base-en-v1.5 \| Local \| 768 \| $0 (compute only) \| Quality + cost balance \|
33	\| text-embedding-3-small \| API \| 1,536 \| $0.02 \| General purpose production \|
34	\| text-embedding-3-large \| API \| 3,072 \| $0.13 \| Premium quality requirements \|
35	\| embed-multilingual-v3.0 \| API \| 1,024 \| $0.10 \| 100+ language support \|
36
37	## Chunking Strategies
38
39	Select chunking strategy based on content type and use case:
40
41	Content Type → Strategy Mapping:
42	- Documentation: Recursive (heading-aware), 800 chars, 100 overlap
43	- Code: Recursive (function-level), 1,000 chars, 100 overlap
44	- Q&A/FAQ: Fixed-size, 500 chars, 50 overlap (precise retrieval)
45	- Legal/Technical: Semantic (large), 1,500 chars, 200 overlap (context preservation)
46	- Blog Posts: Semantic (paragraph), 1,000 chars, 100 overlap
47	- Academic Papers: Recursive (section-aware), 1,200 chars, 150 overlap
48
49	For detailed chunking patterns, decision trees, and implementation guidance, see `references/chunking-strategies.md`.
50
51	Quick Start with CLI:
52	```bash
53	python scripts/chunk_document.py \
54	--input document.txt \
55	--content-type markdown \
56	--chunk-size 800 \
57	--overlap 100 \
58	--output chunks.jsonl
59	```
60
61	## Caching Implementation
62
63	Achieve 80-90% cost reduction through content-addressable caching.
64
65	Caching Architecture by Query Volume:
66	- <10K queries/month: In-memory cache (Python `lru_cache`)
67	- 10K-100K queries/month: Redis (fast, TTL-based expiration)
68	- 100K-1M queries/month: Redis (hot) + PostgreSQL (warm)
69	- >1M queries/month: Multi-tier (Redis + PostgreSQL + S3)
70
71	Production Caching with Redis:
72	```bash
73	# Embed documents with caching enabled
74	python scripts/cached_embedder.py \
75	--model text-embedding-3-small \
76	--input documents.jsonl \
77	--output embeddings.npy \
78	--cache-backend redis \
79	--cache-ttl 2592000 # 30 days
80	```
81
82	Caching ROI Example:
83	- 50,000 document chunks
84	- 20% duplicate content
85	- Without caching: $0.50 API cost
86	- With caching (60% hit rate): $0.20 API cost
87	- Savings: 60% ($0.30)
88
89	## Dimensionality Trade-offs
90
91	Balance storage, search speed, and quality:
92
93	\| Dimensions \| Storage (1M vectors) \| Search Speed (p95) \| Quality \| Use Case \|
94	\|-----------\|---------------------\|-------------------\|---------\|----------\|
95	\| 384 \| 1.5 GB \| 10ms \| Good \| Large-scale search \|
96	\| 768 \| 3 GB \| 15ms \| High \| General purpose RAG \|
97	\| 1,536 \| 6 GB \| 25ms \| Very High \| High-quality retrieval \|
98	\| 3,072 \| 12 GB \| 40ms \| Highest \| Premium applications \|
99
100	Key Insight: For most RAG applications, 768 dimensions (BGE-base-en-v1.5 local or equivalent) provides the best quality/cost/speed balance.
101
102	## Batch Processing Optimization
103
104	Maximize throughput for large-scale ingestion:
105
106	OpenAI API:
107	- Batch up to 2,048 inputs per request
108	- Implement rate limiting (tier-dependent: 500-5,000 RPM)
109	- Use parallel requests with backoff on rate limits
110
111	Local Models (sentence-transformers):
112	- GPU acceleration (CUDA, MPS for Apple Silicon)
113	- Batch size tuning (32-128 based on GPU memory)
114	- Multi-GPU support for maximum throughput
115
116	Expected Throughput:
117	- OpenAI API: 1,000-5,000 texts/minute (rate limit dependent)
118	- Local GPU (RTX 3090): 5,000-10,000 texts/minute
119	- Local CPU: 100-500 texts/minute
120
121	## Performance Monitoring
122
123	Track key metrics for optimization:
124
125	Critical Metrics:
126	-

People who liked this also viewed

6 picks

Type

embedding-optimization

55installs59forks

View on GitHub

How to install embedding-optimization?

ancoleman/ai-design-components/embedding-optimization

$npx -y skills add ancoleman/ai-design-components --skill embedding-optimization

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Embedding Optimization
2
3	Optimize embedding generation for cost, performance, and quality in RAG and semantic search systems.
4
5	## When to Use This Skill
6
7	Trigger this skill when:
8	- Building RAG (Retrieval Augmented Generation) systems
9	- Implementing semantic search or similarity detection
10	- Optimizing embedding API costs (reducing by 70-90%)
11	- Improving document retrieval quality through better chunking
12	- Processing large document corpora (thousands to millions of documents)
13	- Selecting between API-based vs. local embedding models
14
15	## Model Selection Framework
16
17	Choose the optimal embedding model based on requirements:
18
19	Quick Recommendations:
20	- Startup/MVP: `all-MiniLM-L6-v2` (local, 384 dims, zero API costs)
21	- Production: `text-embedding-3-small` (API, 1,536 dims, balanced quality/cost)
22	- High Quality: `text-embedding-3-large` (API, 3,072 dims, premium)
23	- Multilingual: `multilingual-e5-base` (local, 768 dims) or Cohere `embed-multilingual-v3.0`
24
25	For detailed decision frameworks including cost comparisons, quality benchmarks, and data privacy considerations, see `references/model-selection-guide.md`.
26
27	Model Comparison Summary:
28
29	\| Model \| Type \| Dimensions \| Cost per 1M tokens \| Best For \|
30	\|-------\|------\|-----------\|-------------------\|----------\|
31	\| all-MiniLM-L6-v2 \| Local \| 384 \| $0 (compute only) \| High volume, tight budgets \|
32	\| BGE-base-en-v1.5 \| Local \| 768 \| $0 (compute only) \| Quality + cost balance \|
33	\| text-embedding-3-small \| API \| 1,536 \| $0.02 \| General purpose production \|
34	\| text-embedding-3-large \| API \| 3,072 \| $0.13 \| Premium quality requirements \|
35	\| embed-multilingual-v3.0 \| API \| 1,024 \| $0.10 \| 100+ language support \|
36
37	## Chunking Strategies
38
39	Select chunking strategy based on content type and use case:
40
41	Content Type → Strategy Mapping:
42	- Documentation: Recursive (heading-aware), 800 chars, 100 overlap
43	- Code: Recursive (function-level), 1,000 chars, 100 overlap
44	- Q&A/FAQ: Fixed-size, 500 chars, 50 overlap (precise retrieval)
45	- Legal/Technical: Semantic (large), 1,500 chars, 200 overlap (context preservation)
46	- Blog Posts: Semantic (paragraph), 1,000 chars, 100 overlap
47	- Academic Papers: Recursive (section-aware), 1,200 chars, 150 overlap
48
49	For detailed chunking patterns, decision trees, and implementation guidance, see `references/chunking-strategies.md`.
50
51	Quick Start with CLI:
52	```bash
53	python scripts/chunk_document.py \
54	--input document.txt \
55	--content-type markdown \
56	--chunk-size 800 \
57	--overlap 100 \
58	--output chunks.jsonl
59	```
60
61	## Caching Implementation
62
63	Achieve 80-90% cost reduction through content-addressable caching.
64
65	Caching Architecture by Query Volume:
66	- <10K queries/month: In-memory cache (Python `lru_cache`)
67	- 10K-100K queries/month: Redis (fast, TTL-based expiration)
68	- 100K-1M queries/month: Redis (hot) + PostgreSQL (warm)
69	- >1M queries/month: Multi-tier (Redis + PostgreSQL + S3)
70
71	Production Caching with Redis:
72	```bash
73	# Embed documents with caching enabled
74	python scripts/cached_embedder.py \
75	--model text-embedding-3-small \
76	--input documents.jsonl \
77	--output embeddings.npy \
78	--cache-backend redis \
79	--cache-ttl 2592000 # 30 days
80	```
81
82	Caching ROI Example:
83	- 50,000 document chunks
84	- 20% duplicate content
85	- Without caching: $0.50 API cost
86	- With caching (60% hit rate): $0.20 API cost
87	- Savings: 60% ($0.30)
88
89	## Dimensionality Trade-offs
90
91	Balance storage, search speed, and quality:
92
93	\| Dimensions \| Storage (1M vectors) \| Search Speed (p95) \| Quality \| Use Case \|
94	\|-----------\|---------------------\|-------------------\|---------\|----------\|
95	\| 384 \| 1.5 GB \| 10ms \| Good \| Large-scale search \|
96	\| 768 \| 3 GB \| 15ms \| High \| General purpose RAG \|
97	\| 1,536 \| 6 GB \| 25ms \| Very High \| High-quality retrieval \|
98	\| 3,072 \| 12 GB \| 40ms \| Highest \| Premium applications \|
99
100	Key Insight: For most RAG applications, 768 dimensions (BGE-base-en-v1.5 local or equivalent) provides the best quality/cost/speed balance.
101
102	## Batch Processing Optimization
103
104	Maximize throughput for large-scale ingestion:
105
106	OpenAI API:
107	- Batch up to 2,048 inputs per request
108	- Implement rate limiting (tier-dependent: 500-5,000 RPM)
109	- Use parallel requests with backoff on rate limits
110
111	Local Models (sentence-transformers):
112	- GPU acceleration (CUDA, MPS for Apple Silicon)
113	- Batch size tuning (32-128 based on GPU memory)
114	- Multi-GPU support for maximum throughput
115
116	Expected Throughput:
117	- OpenAI API: 1,000-5,000 texts/minute (rate limit dependent)
118	- Local GPU (RTX 3090): 5,000-10,000 texts/minute
119	- Local CPU: 100-500 texts/minute
120
121	## Performance Monitoring
122
123	Track key metrics for optimization:
124
125	Critical Metrics:
126	-

People who liked this also viewed

6 picks

Type