vllm

Name: vllm
Author: AlexAI-MCP

132stars24forks

How to install vllm?

alexai-mcp/hermes-ccc/vllm

$npx -y skills add AlexAI-MCP/hermes-CCC --skill vllm

Installs into the current project.

Summary

Deploy and serve LLMs with vLLM — OpenAI-compatible inference server with PagedAttention, continuous batching, and quantization support.

Files · 1

View on GitHub

SKILL.md

1	# vLLM
2
3	## Purpose
4
5	- Use this skill to deploy local or remote LLM inference with `vllm`.
6	- Prefer it when you need OpenAI-compatible serving, high throughput, and modern GPU utilization.
7	- vLLM is strongest for decoder-only chat and completion models.
8	- It is a good default for production inference when latency and token throughput matter.
9
10	## Install
11
12	- Install from PyPI:
13
14	```bash
15	pip install vllm
16	```
17
18	- Verify the install:
19
20	```bash
21	python -c "import vllm; print(vllm.__version__)"
22	```
23
24	- Match your CUDA, NVIDIA driver, and PyTorch stack before deploying on GPUs.
25	- If deployment is containerized, prefer pinning a known-good image or package version.
26
27	## What vLLM Gives You
28
29	- OpenAI-compatible HTTP API for chat and completion workflows
30	- PagedAttention for efficient KV cache memory usage
31	- Continuous batching for higher aggregate throughput
32	- Tensor parallelism for multi-GPU serving
33	- Quantization support for lower memory footprints
34	- Streaming token responses
35	- Good support for major open-weight model families
36
37	## Common Models
38
39	- `meta-llama/Llama-3.1-8B-Instruct`
40	- `meta-llama/Llama-3.1-70B-Instruct`
41	- `Qwen/Qwen2.5-7B-Instruct`
42	- `Qwen/Qwen2.5-14B-Instruct`
43	- `mistralai/Mistral-7B-Instruct-v0.3`
44	- `microsoft/Phi-3-medium-4k-instruct`
45	- `google/gemma-2-9b-it`
46
47	## Basic Serve
48
49	- Minimal OpenAI-compatible server:
50
51	```bash
52	python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct
53	```
54
55	- By default this serves on port `8000`.
56	- The API base path is `http://localhost:8000/v1`.
57	- Health check endpoint:
58
59	```bash
60	curl http://localhost:8000/health
61	```
62
63	## Key Flags
64
65	- `--port`: bind a non-default server port
66	- `--tensor-parallel-size`: split one model across multiple GPUs
67	- `--gpu-memory-utilization`: cap how much GPU RAM vLLM should attempt to use
68	- `--max-model-len`: set maximum sequence length for inference
69	- `--quantization`: load quantized weights when supported
70
71	## Serve Examples
72
73	- Serve on port `8080`:
74
75	```bash
76	python -m vllm.entrypoints.openai.api_server \
77	--model meta-llama/Llama-3.1-8B-Instruct \
78	--port 8080
79	```
80
81	- Raise the context window and tune memory usage:
82
83	```bash
84	python -m vllm.entrypoints.openai.api_server \
85	--model Qwen/Qwen2.5-7B-Instruct \
86	--max-model-len 32768 \
87	--gpu-memory-utilization 0.92
88	```
89
90	- Use four GPUs with tensor parallelism:
91
92	```bash
93	python -m vllm.entrypoints.openai.api_server \
94	--model meta-llama/Llama-3.1-70B-Instruct \
95	--tensor-parallel-size 4
96	```
97
98	## Quantization
99
100	- vLLM commonly works with these quantization paths:
101	- `awq`
102	- `gptq`
103	- `bitsandbytes`
104	- `fp8`
105
106	- Example with AWQ:
107
108	```bash
109	python -m vllm.entrypoints.openai.api_server \
110	--model Qwen/Qwen2.5-7B-Instruct-AWQ \
111	--quantization awq
112	```
113
114	- Example with GPTQ:
115
116	```bash
117	python -m vllm.entrypoints.openai.api_server \
118	--model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
119	--quantization gptq
120	```
121
122	- Example with FP8:
123
124	```bash
125	python -m vllm.entrypoints.openai.api_server \
126	--model meta-llama/Llama-3.1-8B-Instruct \
127	--quantization fp8
128	```
129
130	- AWQ and GPTQ reduce VRAM pressure at some accuracy or compatibility cost.
131	- `bitsandbytes` is useful when using 4-bit or 8-bit HF-compatible flows.
132	- FP8 is hardware-sensitive and best validated on your actual deployment target.
133
134	## Call Through the OpenAI Client
135
136	- vLLM exposes an OpenAI-style API, so the standard OpenAI Python client is a common fit.
137	- Point the client at the local base URL:
138
139	```python
140	from openai import OpenAI
141
142	client = OpenAI(
143	api_key="dummy",
144	base_url="http://localhost:8000/v1",
145	)
146
147	response = client.chat.completions.create(
148	model="meta-llama/Llama-3.1-8B-Instruct",
149	messages=[
150	{"role": "system", "content": "You are a concise assistant."},
151	{"role": "user", "content": "Summarize why continuous batching matters."},
152	],
153	temperature=0.2,
154	max_tokens=256,
155	)
156
157	print(response.choices[0].message.content)
158	```
159
160	- `base_url='http://localhost:8000/v1'` is the key integration setting.
161	- Many SDK-based applications can switch from OpenAI-hosted inference to vLLM with only the base URL and model name changed.
162
163	## Curl Chat Example
164
165	```bash
166	curl http://localhost:8000/v1/chat/completions \
167	-H "Content-Type: application/json" \
168	-d '{
169	"model": "meta-llama/Llama-3.1-8B-Instruct",
170	"messages": [
171	{"role": "system", "content": "You are concise."},
172	{"role": "user", "content": "Explain PagedAttention in two sentences."}
173	],
174	"temperature": 0.2,
175	"max_tokens": 128
176	}'
177	```
178
179	## Async Engine Usage in Python
180
181	- Use the async engine when embedding vLLM inside a Python application instead of only exposing the HTTP

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

vllm

132stars24forks

View on GitHub

How to install vllm?

alexai-mcp/hermes-ccc/vllm

$npx -y skills add AlexAI-MCP/hermes-CCC --skill vllm

Installs into the current project.

Summary

Deploy and serve LLMs with vLLM — OpenAI-compatible inference server with PagedAttention, continuous batching, and quantization support.

Files · 1

View on GitHub

SKILL.md

1	# vLLM
2
3	## Purpose
4
5	- Use this skill to deploy local or remote LLM inference with `vllm`.
6	- Prefer it when you need OpenAI-compatible serving, high throughput, and modern GPU utilization.
7	- vLLM is strongest for decoder-only chat and completion models.
8	- It is a good default for production inference when latency and token throughput matter.
9
10	## Install
11
12	- Install from PyPI:
13
14	```bash
15	pip install vllm
16	```
17
18	- Verify the install:
19
20	```bash
21	python -c "import vllm; print(vllm.__version__)"
22	```
23
24	- Match your CUDA, NVIDIA driver, and PyTorch stack before deploying on GPUs.
25	- If deployment is containerized, prefer pinning a known-good image or package version.
26
27	## What vLLM Gives You
28
29	- OpenAI-compatible HTTP API for chat and completion workflows
30	- PagedAttention for efficient KV cache memory usage
31	- Continuous batching for higher aggregate throughput
32	- Tensor parallelism for multi-GPU serving
33	- Quantization support for lower memory footprints
34	- Streaming token responses
35	- Good support for major open-weight model families
36
37	## Common Models
38
39	- `meta-llama/Llama-3.1-8B-Instruct`
40	- `meta-llama/Llama-3.1-70B-Instruct`
41	- `Qwen/Qwen2.5-7B-Instruct`
42	- `Qwen/Qwen2.5-14B-Instruct`
43	- `mistralai/Mistral-7B-Instruct-v0.3`
44	- `microsoft/Phi-3-medium-4k-instruct`
45	- `google/gemma-2-9b-it`
46
47	## Basic Serve
48
49	- Minimal OpenAI-compatible server:
50
51	```bash
52	python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-8B-Instruct
53	```
54
55	- By default this serves on port `8000`.
56	- The API base path is `http://localhost:8000/v1`.
57	- Health check endpoint:
58
59	```bash
60	curl http://localhost:8000/health
61	```
62
63	## Key Flags
64
65	- `--port`: bind a non-default server port
66	- `--tensor-parallel-size`: split one model across multiple GPUs
67	- `--gpu-memory-utilization`: cap how much GPU RAM vLLM should attempt to use
68	- `--max-model-len`: set maximum sequence length for inference
69	- `--quantization`: load quantized weights when supported
70
71	## Serve Examples
72
73	- Serve on port `8080`:
74
75	```bash
76	python -m vllm.entrypoints.openai.api_server \
77	--model meta-llama/Llama-3.1-8B-Instruct \
78	--port 8080
79	```
80
81	- Raise the context window and tune memory usage:
82
83	```bash
84	python -m vllm.entrypoints.openai.api_server \
85	--model Qwen/Qwen2.5-7B-Instruct \
86	--max-model-len 32768 \
87	--gpu-memory-utilization 0.92
88	```
89
90	- Use four GPUs with tensor parallelism:
91
92	```bash
93	python -m vllm.entrypoints.openai.api_server \
94	--model meta-llama/Llama-3.1-70B-Instruct \
95	--tensor-parallel-size 4
96	```
97
98	## Quantization
99
100	- vLLM commonly works with these quantization paths:
101	- `awq`
102	- `gptq`
103	- `bitsandbytes`
104	- `fp8`
105
106	- Example with AWQ:
107
108	```bash
109	python -m vllm.entrypoints.openai.api_server \
110	--model Qwen/Qwen2.5-7B-Instruct-AWQ \
111	--quantization awq
112	```
113
114	- Example with GPTQ:
115
116	```bash
117	python -m vllm.entrypoints.openai.api_server \
118	--model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ \
119	--quantization gptq
120	```
121
122	- Example with FP8:
123
124	```bash
125	python -m vllm.entrypoints.openai.api_server \
126	--model meta-llama/Llama-3.1-8B-Instruct \
127	--quantization fp8
128	```
129
130	- AWQ and GPTQ reduce VRAM pressure at some accuracy or compatibility cost.
131	- `bitsandbytes` is useful when using 4-bit or 8-bit HF-compatible flows.
132	- FP8 is hardware-sensitive and best validated on your actual deployment target.
133
134	## Call Through the OpenAI Client
135
136	- vLLM exposes an OpenAI-style API, so the standard OpenAI Python client is a common fit.
137	- Point the client at the local base URL:
138
139	```python
140	from openai import OpenAI
141
142	client = OpenAI(
143	api_key="dummy",
144	base_url="http://localhost:8000/v1",
145	)
146
147	response = client.chat.completions.create(
148	model="meta-llama/Llama-3.1-8B-Instruct",
149	messages=[
150	{"role": "system", "content": "You are a concise assistant."},
151	{"role": "user", "content": "Summarize why continuous batching matters."},
152	],
153	temperature=0.2,
154	max_tokens=256,
155	)
156
157	print(response.choices[0].message.content)
158	```
159
160	- `base_url='http://localhost:8000/v1'` is the key integration setting.
161	- Many SDK-based applications can switch from OpenAI-hosted inference to vLLM with only the base URL and model name changed.
162
163	## Curl Chat Example
164
165	```bash
166	curl http://localhost:8000/v1/chat/completions \
167	-H "Content-Type: application/json" \
168	-d '{
169	"model": "meta-llama/Llama-3.1-8B-Instruct",
170	"messages": [
171	{"role": "system", "content": "You are concise."},
172	{"role": "user", "content": "Explain PagedAttention in two sentences."}
173	],
174	"temperature": 0.2,
175	"max_tokens": 128
176	}'
177	```
178
179	## Async Engine Usage in Python
180
181	- Use the async engine when embedding vLLM inside a Python application instead of only exposing the HTTP

People who liked this also viewed

6 picks

Type