llm-cost-optimization

Name: llm-cost-optimization
Author: BagelHole

42stars4forks

How to install llm-cost-optimization?

bagelhole/devops-security-agent-skills/llm-cost-optimization

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-cost-optimization

Installs into the current project.

Summary

Reduce LLM API and infrastructure costs through model selection, prompt caching, batching, caching, quantization, and self-hosting strategies. Track spend by team and model, set budgets, and implement cost-aware routing.

Files · 1

View on GitHub

SKILL.md

1	# LLM Cost Optimization
2
3	Cut LLM costs by 50–90% with the right combination of caching, model selection, prompt optimization, and self-hosting.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- LLM API spend is growing faster than revenue
9	- You need to attribute AI costs to teams, products, or customers
10	- Implementing caching to avoid redundant LLM calls
11	- Deciding when to switch from API providers to self-hosted models
12	- Optimizing prompt length without sacrificing quality
13
14	## Cost Levers by Impact
15
16	\| Strategy \| Typical Savings \| Effort \|
17	\|----------\|-----------------\|--------\|
18	\| Semantic caching \| 20–50% \| Low \|
19	\| Model right-sizing \| 30–70% \| Low \|
20	\| Prompt compression \| 10–30% \| Medium \|
21	\| Provider caching (prompt cache) \| 10–25% \| Low \|
22	\| Batching offline workloads \| 50% (Batch API) \| Medium \|
23	\| Self-hosting 7–8B models \| 80–95% at scale \| High \|
24	\| Quantization \| 30–50% VRAM cost \| Medium \|
25
26	## Track Costs First
27
28	```python
29	# Use LiteLLM's cost tracking (automatic per-model pricing)
30	import litellm
31
32	response = litellm.completion(
33	model="gpt-4o-mini",
34	messages=[{"role": "user", "content": "Hello"}],
35	)
36	cost = litellm.completion_cost(response)
37	print(f"Cost: ${cost:.6f}")
38
39	# Add custom cost callbacks
40	def log_cost(kwargs, completion_response, start_time, end_time):
41	cost = kwargs.get("response_cost", 0)
42	model = kwargs.get("model")
43	user = kwargs.get("user")
44	# Send to your analytics DB
45	db.record_cost(user=user, model=model, cost=cost)
46
47	litellm.success_callback = [log_cost]
48	```
49
50	## Model Right-Sizing
51
52	```python
53	# Route by task complexity — don't use GPT-4o for everything
54	def get_model_for_task(task_type: str) -> str:
55	routing = {
56	"classification": "gpt-4o-mini", # ~30× cheaper than gpt-4o
57	"summarization": "gpt-4o-mini",
58	"extraction": "gpt-4o-mini",
59	"simple_qa": "gpt-4o-mini",
60	"complex_reasoning": "gpt-4o",
61	"code_generation": "claude-sonnet-4-6",
62	"creative_writing": "claude-opus-4-6",
63	}
64	return routing.get(task_type, "gpt-4o-mini")
65
66	# Cost comparison (per 1M tokens, 2025 approx.)
67	# gpt-4o-mini: input $0.15 / output $0.60
68	# gpt-4o: input $2.50 / output $10.00
69	# claude-sonnet-4-6: input $3.00 / output $15.00
70	# llama-3.1-8b (self): ~$0.05–0.10 all-in (GPU amortized)
71	```
72
73	## Prompt Caching (Provider-Side)
74
75	```python
76	# Anthropic — cache long system prompts (saves 90% on cached tokens)
77	import anthropic
78
79	client = anthropic.Anthropic()
80
81	response = client.messages.create(
82	model="claude-sonnet-4-6",
83	max_tokens=1024,
84	system=[
85	{
86	"type": "text",
87	"text": "You are a helpful assistant.",
88	},
89	{
90	"type": "text",
91	"text": open("large-context.txt").read(), # large doc
92	"cache_control": {"type": "ephemeral"}, # cache this!
93	}
94	],
95	messages=[{"role": "user", "content": "Summarize the key points."}],
96	)
97	# First call: full price. Subsequent calls: 90% discount on cached part.
98	print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
99
100	# OpenAI — prompt caching is automatic for repeated prefixes >1024 tokens
101	# No code change needed; check usage.prompt_tokens_details.cached_tokens
102	```
103
104	## Batching with OpenAI Batch API (50% Discount)
105
106	```python
107	import json
108	from openai import OpenAI
109
110	client = OpenAI()
111
112	# Prepare batch requests
113	requests = [
114	{
115	"custom_id": f"task-{i}",
116	"method": "POST",
117	"url": "/v1/chat/completions",
118	"body": {
119	"model": "gpt-4o-mini",
120	"messages": [{"role": "user", "content": f"Classify: {text}"}],
121	"max_tokens": 50,
122	}
123	}
124	for i, text in enumerate(texts)
125	]
126
127	# Write JSONL file
128	with open("batch.jsonl", "w") as f:
129	for req in requests:
130	f.write(json.dumps(req) + "\n")
131
132	# Upload and create batch
133	batch_file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
134	batch = client.batches.create(
135	input_file_id=batch_file.id,
136	endpoint="/v1/chat/completions",
137	completion_window="24h",
138	)
139	print(f"Batch ID: {batch.id}") # poll status with client.batches.retrieve(batch.id)
140	```
141
142	## Semantic Caching
143
144	```python
145	import hashlib
146	import json
147	import redis
148	import numpy as np
149	from sentence_transformers import SentenceTransformer
150
151	r = redis.Redis(host="localhost", port=6379)
152	embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")
153
154	SIMILARITY_THRESHOLD = 0.92
155	CACHE_TTL = 3600 * 24 # 24 hours
156
157	def cached_llm_call(prompt: str, llm_fn) -> str:
158	# 1. Exact match (free)
159	exact_key = f"exact:{hashlib.sha256(prompt.encode()).hex

People who liked this also viewed

6 picks

Type

llm-cost-optimization

42stars4forks

View on GitHub

How to install llm-cost-optimization?

bagelhole/devops-security-agent-skills/llm-cost-optimization

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill llm-cost-optimization

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# LLM Cost Optimization
2
3	Cut LLM costs by 50–90% with the right combination of caching, model selection, prompt optimization, and self-hosting.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- LLM API spend is growing faster than revenue
9	- You need to attribute AI costs to teams, products, or customers
10	- Implementing caching to avoid redundant LLM calls
11	- Deciding when to switch from API providers to self-hosted models
12	- Optimizing prompt length without sacrificing quality
13
14	## Cost Levers by Impact
15
16	\| Strategy \| Typical Savings \| Effort \|
17	\|----------\|-----------------\|--------\|
18	\| Semantic caching \| 20–50% \| Low \|
19	\| Model right-sizing \| 30–70% \| Low \|
20	\| Prompt compression \| 10–30% \| Medium \|
21	\| Provider caching (prompt cache) \| 10–25% \| Low \|
22	\| Batching offline workloads \| 50% (Batch API) \| Medium \|
23	\| Self-hosting 7–8B models \| 80–95% at scale \| High \|
24	\| Quantization \| 30–50% VRAM cost \| Medium \|
25
26	## Track Costs First
27
28	```python
29	# Use LiteLLM's cost tracking (automatic per-model pricing)
30	import litellm
31
32	response = litellm.completion(
33	model="gpt-4o-mini",
34	messages=[{"role": "user", "content": "Hello"}],
35	)
36	cost = litellm.completion_cost(response)
37	print(f"Cost: ${cost:.6f}")
38
39	# Add custom cost callbacks
40	def log_cost(kwargs, completion_response, start_time, end_time):
41	cost = kwargs.get("response_cost", 0)
42	model = kwargs.get("model")
43	user = kwargs.get("user")
44	# Send to your analytics DB
45	db.record_cost(user=user, model=model, cost=cost)
46
47	litellm.success_callback = [log_cost]
48	```
49
50	## Model Right-Sizing
51
52	```python
53	# Route by task complexity — don't use GPT-4o for everything
54	def get_model_for_task(task_type: str) -> str:
55	routing = {
56	"classification": "gpt-4o-mini", # ~30× cheaper than gpt-4o
57	"summarization": "gpt-4o-mini",
58	"extraction": "gpt-4o-mini",
59	"simple_qa": "gpt-4o-mini",
60	"complex_reasoning": "gpt-4o",
61	"code_generation": "claude-sonnet-4-6",
62	"creative_writing": "claude-opus-4-6",
63	}
64	return routing.get(task_type, "gpt-4o-mini")
65
66	# Cost comparison (per 1M tokens, 2025 approx.)
67	# gpt-4o-mini: input $0.15 / output $0.60
68	# gpt-4o: input $2.50 / output $10.00
69	# claude-sonnet-4-6: input $3.00 / output $15.00
70	# llama-3.1-8b (self): ~$0.05–0.10 all-in (GPU amortized)
71	```
72
73	## Prompt Caching (Provider-Side)
74
75	```python
76	# Anthropic — cache long system prompts (saves 90% on cached tokens)
77	import anthropic
78
79	client = anthropic.Anthropic()
80
81	response = client.messages.create(
82	model="claude-sonnet-4-6",
83	max_tokens=1024,
84	system=[
85	{
86	"type": "text",
87	"text": "You are a helpful assistant.",
88	},
89	{
90	"type": "text",
91	"text": open("large-context.txt").read(), # large doc
92	"cache_control": {"type": "ephemeral"}, # cache this!
93	}
94	],
95	messages=[{"role": "user", "content": "Summarize the key points."}],
96	)
97	# First call: full price. Subsequent calls: 90% discount on cached part.
98	print(f"Cache read tokens: {response.usage.cache_read_input_tokens}")
99
100	# OpenAI — prompt caching is automatic for repeated prefixes >1024 tokens
101	# No code change needed; check usage.prompt_tokens_details.cached_tokens
102	```
103
104	## Batching with OpenAI Batch API (50% Discount)
105
106	```python
107	import json
108	from openai import OpenAI
109
110	client = OpenAI()
111
112	# Prepare batch requests
113	requests = [
114	{
115	"custom_id": f"task-{i}",
116	"method": "POST",
117	"url": "/v1/chat/completions",
118	"body": {
119	"model": "gpt-4o-mini",
120	"messages": [{"role": "user", "content": f"Classify: {text}"}],
121	"max_tokens": 50,
122	}
123	}
124	for i, text in enumerate(texts)
125	]
126
127	# Write JSONL file
128	with open("batch.jsonl", "w") as f:
129	for req in requests:
130	f.write(json.dumps(req) + "\n")
131
132	# Upload and create batch
133	batch_file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
134	batch = client.batches.create(
135	input_file_id=batch_file.id,
136	endpoint="/v1/chat/completions",
137	completion_window="24h",
138	)
139	print(f"Batch ID: {batch.id}") # poll status with client.batches.retrieve(batch.id)
140	```
141
142	## Semantic Caching
143
144	```python
145	import hashlib
146	import json
147	import redis
148	import numpy as np
149	from sentence_transformers import SentenceTransformer
150
151	r = redis.Redis(host="localhost", port=6379)
152	embed_model = SentenceTransformer("BAAI/bge-small-en-v1.5")
153
154	SIMILARITY_THRESHOLD = 0.92
155	CACHE_TTL = 3600 * 24 # 24 hours
156
157	def cached_llm_call(prompt: str, llm_fn) -> str:
158	# 1. Exact match (free)
159	exact_key = f"exact:{hashlib.sha256(prompt.encode()).hex

People who liked this also viewed

6 picks

Type