model-serving

Name: model-serving
Author: ancoleman

43installs59forks

How to install model-serving?

ancoleman/ai-design-components/model-serving

$npx -y skills add ancoleman/ai-design-components --skill model-serving

Installs into the current project.

Summary

LLM and ML model deployment for inference. Use when serving models in production, building AI APIs, or optimizing inference. Covers vLLM (LLM serving), TensorRT-LLM (GPU optimization), Ollama (local), BentoML (ML deployment), Triton (multi-model), LangChain (orchestration), Llama

Files · 1

View on GitHub

SKILL.md

1	# Model Serving
2
3	## Purpose
4
5	Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.
6
7	## When to Use
8
9	- Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
10	- Building AI APIs with streaming responses
11	- Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
12	- Implementing RAG pipelines with vector databases
13	- Optimizing inference throughput and latency
14	- Integrating LLM serving with frontend chat interfaces
15
16	## Model Serving Selection
17
18	### LLM Serving Engines
19
20	vLLM (Recommended Primary)
21	- PagedAttention memory management (20-30x throughput improvement)
22	- Continuous batching for dynamic request handling
23	- OpenAI-compatible API endpoints
24	- Use for: Most self-hosted LLM deployments
25
26	TensorRT-LLM
27	- Maximum GPU efficiency (2-8x faster than vLLM)
28	- Requires model conversion and optimization
29	- Use for: Production workloads needing absolute maximum throughput
30
31	Ollama
32	- Local development without GPUs
33	- Simple CLI interface
34	- Use for: Prototyping, laptop development, educational purposes
35
36	Decision Framework:
37	```
38	Self-hosted LLM deployment needed?
39	├─ Yes, need maximum throughput → vLLM
40	├─ Yes, need absolute max GPU efficiency → TensorRT-LLM
41	├─ Yes, local development only → Ollama
42	└─ No, use managed API (OpenAI, Anthropic) → No serving layer needed
43	```
44
45	### ML Model Serving (Non-LLM)
46
47	BentoML (Recommended)
48	- Python-native, easy deployment
49	- Adaptive batching for throughput
50	- Multi-framework support (scikit-learn, PyTorch, XGBoost)
51	- Use for: Most traditional ML model deployments
52
53	Triton Inference Server
54	- Multi-model serving on same GPU
55	- Model ensembles (chain multiple models)
56	- Use for: NVIDIA GPU optimization, serving 10+ models
57
58	### LLM Orchestration
59
60	LangChain
61	- General-purpose workflows, agents, RAG
62	- 100+ integrations (LLMs, vector DBs, tools)
63	- Use for: Most RAG and agent applications
64
65	LlamaIndex
66	- RAG-focused with advanced retrieval strategies
67	- 100+ data connectors (PDF, Notion, web)
68	- Use for: RAG is primary use case
69
70	## Quick Start Examples
71
72	### vLLM Server Setup
73
74	```bash
75	# Install
76	pip install vllm
77
78	# Serve a model (OpenAI-compatible API)
79	vllm serve meta-llama/Llama-3.1-8B-Instruct \
80	--dtype auto \
81	--max-model-len 4096 \
82	--gpu-memory-utilization 0.9 \
83	--port 8000
84	```
85
86	Key Parameters:
87	- `--dtype`: Model precision (auto, float16, bfloat16)
88	- `--max-model-len`: Context window size
89	- `--gpu-memory-utilization`: GPU memory fraction (0.8-0.95)
90	- `--tensor-parallel-size`: Number of GPUs for model parallelism
91
92	### Streaming Responses (SSE Pattern)
93
94	Backend (FastAPI):
95	```python
96	from fastapi import FastAPI
97	from fastapi.responses import StreamingResponse
98	from openai import OpenAI
99	import json
100
101	app = FastAPI()
102	client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
103
104	@app.post("/chat/stream")
105	async def chat_stream(message: str):
106	async def generate():
107	stream = client.chat.completions.create(
108	model="meta-llama/Llama-3.1-8B-Instruct",
109	messages=[{"role": "user", "content": message}],
110	stream=True,
111	max_tokens=512
112	)
113
114	for chunk in stream:
115	if chunk.choices[0].delta.content:
116	token = chunk.choices[0].delta.content
117	yield f"data: {json.dumps({'token': token})}\n\n"
118
119	yield f"data: {json.dumps({'done': True})}\n\n"
120
121	return StreamingResponse(
122	generate(),
123	media_type="text/event-stream",
124	headers={"Cache-Control": "no-cache"}
125	)
126	```
127
128	Frontend (React):
129	```typescript
130	// Integration with ai-chat skill
131	const sendMessage = async (message: string) => {
132	const response = await fetch('/chat/stream', {
133	method: 'POST',
134	headers: { 'Content-Type': 'application/json' },
135	body: JSON.stringify({ message })
136	})
137
138	const reader = response.body!.getReader()
139	const decoder = new TextDecoder()
140
141	while (true) {
142	const { done, value } = await reader.read()
143	if (done) break
144
145	const chunk = decoder.decode(value)
146	const lines = chunk.split('\n\n')
147
148	for (const line of lines) {
149	if (line.startsWith('data: ')) {
150	const data = JSON.parse(line.slice(6))
151	if (data.token) {
152	setResponse(prev => prev + data.token)
153	}
154	}
155	}
156	}
157	}
158	```
159
160	### BentoML Service
161
162	```python
163	import bentoml
164	from bentoml.io import JSON
165	import numpy as np
166
167	@bentoml.service(
168	resources={"cpu": "2", "memory": "4Gi"},
169	traffic={"timeout": 10}
170	)
171	cla

People who liked this also viewed

6 picks

Type

model-serving

43installs59forks

View on GitHub

How to install model-serving?

ancoleman/ai-design-components/model-serving

$npx -y skills add ancoleman/ai-design-components --skill model-serving

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model Serving
2
3	## Purpose
4
5	Deploy LLM and ML models for production inference with optimized serving engines, streaming response patterns, and orchestration frameworks. Focuses on self-hosted model serving, GPU optimization, and integration with frontend applications.
6
7	## When to Use
8
9	- Deploying LLMs for production (self-hosted Llama, Mistral, Qwen)
10	- Building AI APIs with streaming responses
11	- Serving traditional ML models (scikit-learn, XGBoost, PyTorch)
12	- Implementing RAG pipelines with vector databases
13	- Optimizing inference throughput and latency
14	- Integrating LLM serving with frontend chat interfaces
15
16	## Model Serving Selection
17
18	### LLM Serving Engines
19
20	vLLM (Recommended Primary)
21	- PagedAttention memory management (20-30x throughput improvement)
22	- Continuous batching for dynamic request handling
23	- OpenAI-compatible API endpoints
24	- Use for: Most self-hosted LLM deployments
25
26	TensorRT-LLM
27	- Maximum GPU efficiency (2-8x faster than vLLM)
28	- Requires model conversion and optimization
29	- Use for: Production workloads needing absolute maximum throughput
30
31	Ollama
32	- Local development without GPUs
33	- Simple CLI interface
34	- Use for: Prototyping, laptop development, educational purposes
35
36	Decision Framework:
37	```
38	Self-hosted LLM deployment needed?
39	├─ Yes, need maximum throughput → vLLM
40	├─ Yes, need absolute max GPU efficiency → TensorRT-LLM
41	├─ Yes, local development only → Ollama
42	└─ No, use managed API (OpenAI, Anthropic) → No serving layer needed
43	```
44
45	### ML Model Serving (Non-LLM)
46
47	BentoML (Recommended)
48	- Python-native, easy deployment
49	- Adaptive batching for throughput
50	- Multi-framework support (scikit-learn, PyTorch, XGBoost)
51	- Use for: Most traditional ML model deployments
52
53	Triton Inference Server
54	- Multi-model serving on same GPU
55	- Model ensembles (chain multiple models)
56	- Use for: NVIDIA GPU optimization, serving 10+ models
57
58	### LLM Orchestration
59
60	LangChain
61	- General-purpose workflows, agents, RAG
62	- 100+ integrations (LLMs, vector DBs, tools)
63	- Use for: Most RAG and agent applications
64
65	LlamaIndex
66	- RAG-focused with advanced retrieval strategies
67	- 100+ data connectors (PDF, Notion, web)
68	- Use for: RAG is primary use case
69
70	## Quick Start Examples
71
72	### vLLM Server Setup
73
74	```bash
75	# Install
76	pip install vllm
77
78	# Serve a model (OpenAI-compatible API)
79	vllm serve meta-llama/Llama-3.1-8B-Instruct \
80	--dtype auto \
81	--max-model-len 4096 \
82	--gpu-memory-utilization 0.9 \
83	--port 8000
84	```
85
86	Key Parameters:
87	- `--dtype`: Model precision (auto, float16, bfloat16)
88	- `--max-model-len`: Context window size
89	- `--gpu-memory-utilization`: GPU memory fraction (0.8-0.95)
90	- `--tensor-parallel-size`: Number of GPUs for model parallelism
91
92	### Streaming Responses (SSE Pattern)
93
94	Backend (FastAPI):
95	```python
96	from fastapi import FastAPI
97	from fastapi.responses import StreamingResponse
98	from openai import OpenAI
99	import json
100
101	app = FastAPI()
102	client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
103
104	@app.post("/chat/stream")
105	async def chat_stream(message: str):
106	async def generate():
107	stream = client.chat.completions.create(
108	model="meta-llama/Llama-3.1-8B-Instruct",
109	messages=[{"role": "user", "content": message}],
110	stream=True,
111	max_tokens=512
112	)
113
114	for chunk in stream:
115	if chunk.choices[0].delta.content:
116	token = chunk.choices[0].delta.content
117	yield f"data: {json.dumps({'token': token})}\n\n"
118
119	yield f"data: {json.dumps({'done': True})}\n\n"
120
121	return StreamingResponse(
122	generate(),
123	media_type="text/event-stream",
124	headers={"Cache-Control": "no-cache"}
125	)
126	```
127
128	Frontend (React):
129	```typescript
130	// Integration with ai-chat skill
131	const sendMessage = async (message: string) => {
132	const response = await fetch('/chat/stream', {
133	method: 'POST',
134	headers: { 'Content-Type': 'application/json' },
135	body: JSON.stringify({ message })
136	})
137
138	const reader = response.body!.getReader()
139	const decoder = new TextDecoder()
140
141	while (true) {
142	const { done, value } = await reader.read()
143	if (done) break
144
145	const chunk = decoder.decode(value)
146	const lines = chunk.split('\n\n')
147
148	for (const line of lines) {
149	if (line.startsWith('data: ')) {
150	const data = JSON.parse(line.slice(6))
151	if (data.token) {
152	setResponse(prev => prev + data.token)
153	}
154	}
155	}
156	}
157	}
158	```
159
160	### BentoML Service
161
162	```python
163	import bentoml
164	from bentoml.io import JSON
165	import numpy as np
166
167	@bentoml.service(
168	resources={"cpu": "2", "memory": "4Gi"},
169	traffic={"timeout": 10}
170	)
171	cla

People who liked this also viewed

6 picks

Type