vllm-bench-serve

Name: vllm-bench-serve
Author: vllm-project

111installs23forks

How to install vllm-bench-serve?

vllm-project/vllm-skills/vllm-bench-serve

$npx -y skills add vllm-project/vllm-skills --skill vllm-bench-serve

Installs into the current project.

Summary

Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve. Supports multiple datasets (random, sharegpt, sonnet, HF), backends (openai, openai-chat, vllm-pooling, embeddings), throughput/latency testing with request-rate control, and result saving. Use when ben

Files · 1

View on GitHub

SKILL.md

1	# vLLM Bench Serve
2
3	Benchmark vLLM or any OpenAI-compatible serving endpoint using the `vllm bench serve` CLI. Measures throughput, latency (TTFT, TPOT), and goodput against configurable request load.
4
5	Reference: [vLLM Bench Serve Documentation](https://docs.vllm.ai/en/latest/cli/bench/serve/)
6
7	## Prerequisites
8
9	- vLLM installed (or any OpenAI-compatible server running)
10	- A vLLM server or API endpoint already serving a model
11	- Python environment with vLLM for the benchmark client
12
13	## Quick Start
14
15	Basic benchmark against local vLLM server (default random dataset, 1000 prompts):
16
17	```bash
18	vllm bench serve \
19	--backend openai-chat \
20	--host 127.0.0.1 \
21	--port 8000 \
22	--model Qwen/Qwen2.5-1.5B-Instruct \
23	--endpoint /v1/chat/completions
24	```
25
26	Save results to JSON:
27
28	```bash
29	vllm bench serve \
30	--backend openai-chat \
31	--host 127.0.0.1 \
32	--port 8000 \
33	--model Qwen/Qwen2.5-1.5B-Instruct \
34	--endpoint /v1/chat/completions \
35	--save-result \
36	--result-dir ./bench-results \
37	--metadata "version=0.6.0" "tp=1"
38	```
39
40	> Note: When using `--backend openai-chat`, you must specify `--endpoint /v1/chat/completions` (default is `/v1/completions`).
41
42	## Core Arguments
43
44	\| Argument \| Default \| Description \|
45	\|----------\|---------\|-------------\|
46	\| `--backend` \| `openai` \| Backend type: `openai`, `openai-chat`, `openai-embeddings`, `vllm`, `vllm-pooling`, `vllm-rerank`, etc. \|
47	\| `--host` \| `127.0.0.1` \| Server host \|
48	\| `--port` \| `8000` \| Server port \|
49	\| `--base-url` \| - \| Alternative: full base URL instead of host:port \|
50	\| `--endpoint` \| `/v1/completions` \| API endpoint; use `/v1/chat/completions` for openai-chat \|
51	\| `--model` \| (from /v1/models) \| Model name \|
52	\| `--num-prompts` \| `1000` \| Number of prompts to process \|
53	\| `--request-rate` \| `inf` \| Requests per second; `inf` = burst all at once \|
54	\| `--max-concurrency` \| - \| Max concurrent requests (caps parallelism) \|
55	\| `--num-warmups` \| `0` \| Warmup requests before measuring \|
56
57	## Datasets
58
59	\| `--dataset-name` \| Use Case \|
60	\|------------------\|----------\|
61	\| `random` \| Synthetic random prompts (default) \|
62	\| `sharegpt` \| ShareGPT conversation format; requires `--dataset-path` \|
63	\| `sonnet` \| Sonnet-style prompts \|
64	\| `hf` \| HuggingFace dataset; requires `--dataset-path` (dataset ID) \|
65	\| `custom` / `custom_mm` \| Custom dataset; requires `--dataset-path` \|
66	\| `prefix_repetition` \| Prefix repetition benchmark \|
67	\| `random-mm` \| Random multimodal (images/videos) \|
68	\| `spec_bench` \| Spec bench dataset \|
69
70	Dataset-specific options (examples):
71
72	```bash
73	# Random: control input/output length
74	--dataset-name random --random-input-len 1024 --random-output-len 128
75
76	# Sonnet defaults: input 550, output 150, prefix 200
77	--dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150
78
79	# HuggingFace dataset
80	--dataset-name hf --dataset-path "lmarena-ai/VisionArena-Chat" --hf-split test
81
82	# General overrides (map to dataset-specific args)
83	--input-len 512 --output-len 256
84	```
85
86	## Load Control
87
88	```bash
89	# Fixed request rate (Poisson process)
90	--request-rate 10
91
92	# More bursty arrivals (gamma distribution, burstiness < 1)
93	--request-rate 10 --burstiness 0.5
94
95	# Ramp-up from low to high RPS
96	--ramp-up-strategy linear --ramp-up-start-rps 1 --ramp-up-end-rps 50
97
98	# Limit concurrency (useful for rate-limited APIs)
99	--max-concurrency 32
100	```
101
102	## Results and Metrics
103
104	\| Argument \| Description \|
105	\|----------\|-------------\|
106	\| `--save-result` \| Save benchmark results to JSON \|
107	\| `--save-detailed` \| Include per-request TTFT, TPOT, errors in JSON \|
108	\| `--append-result` \| Append to existing result file \|
109	\| `--result-dir` \| Directory for result files \|
110	\| `--result-filename` \| Custom filename (default: `{label}-{request_rate}qps-{model}-{timestamp}.json`) \|
111	\| `--percentile-metrics` \| Metrics for percentiles: `ttft`, `tpot`, `itl`, `e2el` (default: `ttft,tpot,itl`) \|
112	\| `--metric-percentiles` \| Percentile values, e.g. `25,50,99` (default: `99`) \|
113	\| `--goodput` \| SLO for goodput: `ttft:500 tpot:50` (ms) \|
114
115	## Sampling Parameters (OpenAI-compatible backends)
116
117	```bash
118	--temperature 0.7 --top-p 0.95 --top-k 50
119	--frequency-penalty 0 --presence-penalty 0 --repetition-penalty 1.0
120	```
121
122	## Common Workflows
123
124	1. Throughput test with random dataset (burst):
125
126	```bash
127	vllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \
128	--model Qwen/Qwen2.5-1.5B-Instruct \
129	--endpoint /v1/chat/completions \
130	--dataset-name random \
131	--num-prompts 500 --random-input-len 512 --random-output-len 128
132	```
133
134	2. Latency test with fixed QPS:
135
136	```bash
137	vllm bench serve --backend openai-chat --host 127.0.0.1 --port

People who liked this also viewed

6 picks

Type

vllm-bench-serve

111installs23forks

View on GitHub

How to install vllm-bench-serve?

vllm-project/vllm-skills/vllm-bench-serve

$npx -y skills add vllm-project/vllm-skills --skill vllm-bench-serve

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# vLLM Bench Serve
2
3	Benchmark vLLM or any OpenAI-compatible serving endpoint using the `vllm bench serve` CLI. Measures throughput, latency (TTFT, TPOT), and goodput against configurable request load.
4
5	Reference: [vLLM Bench Serve Documentation](https://docs.vllm.ai/en/latest/cli/bench/serve/)
6
7	## Prerequisites
8
9	- vLLM installed (or any OpenAI-compatible server running)
10	- A vLLM server or API endpoint already serving a model
11	- Python environment with vLLM for the benchmark client
12
13	## Quick Start
14
15	Basic benchmark against local vLLM server (default random dataset, 1000 prompts):
16
17	```bash
18	vllm bench serve \
19	--backend openai-chat \
20	--host 127.0.0.1 \
21	--port 8000 \
22	--model Qwen/Qwen2.5-1.5B-Instruct \
23	--endpoint /v1/chat/completions
24	```
25
26	Save results to JSON:
27
28	```bash
29	vllm bench serve \
30	--backend openai-chat \
31	--host 127.0.0.1 \
32	--port 8000 \
33	--model Qwen/Qwen2.5-1.5B-Instruct \
34	--endpoint /v1/chat/completions \
35	--save-result \
36	--result-dir ./bench-results \
37	--metadata "version=0.6.0" "tp=1"
38	```
39
40	> Note: When using `--backend openai-chat`, you must specify `--endpoint /v1/chat/completions` (default is `/v1/completions`).
41
42	## Core Arguments
43
44	\| Argument \| Default \| Description \|
45	\|----------\|---------\|-------------\|
46	\| `--backend` \| `openai` \| Backend type: `openai`, `openai-chat`, `openai-embeddings`, `vllm`, `vllm-pooling`, `vllm-rerank`, etc. \|
47	\| `--host` \| `127.0.0.1` \| Server host \|
48	\| `--port` \| `8000` \| Server port \|
49	\| `--base-url` \| - \| Alternative: full base URL instead of host:port \|
50	\| `--endpoint` \| `/v1/completions` \| API endpoint; use `/v1/chat/completions` for openai-chat \|
51	\| `--model` \| (from /v1/models) \| Model name \|
52	\| `--num-prompts` \| `1000` \| Number of prompts to process \|
53	\| `--request-rate` \| `inf` \| Requests per second; `inf` = burst all at once \|
54	\| `--max-concurrency` \| - \| Max concurrent requests (caps parallelism) \|
55	\| `--num-warmups` \| `0` \| Warmup requests before measuring \|
56
57	## Datasets
58
59	\| `--dataset-name` \| Use Case \|
60	\|------------------\|----------\|
61	\| `random` \| Synthetic random prompts (default) \|
62	\| `sharegpt` \| ShareGPT conversation format; requires `--dataset-path` \|
63	\| `sonnet` \| Sonnet-style prompts \|
64	\| `hf` \| HuggingFace dataset; requires `--dataset-path` (dataset ID) \|
65	\| `custom` / `custom_mm` \| Custom dataset; requires `--dataset-path` \|
66	\| `prefix_repetition` \| Prefix repetition benchmark \|
67	\| `random-mm` \| Random multimodal (images/videos) \|
68	\| `spec_bench` \| Spec bench dataset \|
69
70	Dataset-specific options (examples):
71
72	```bash
73	# Random: control input/output length
74	--dataset-name random --random-input-len 1024 --random-output-len 128
75
76	# Sonnet defaults: input 550, output 150, prefix 200
77	--dataset-name sonnet --sonnet-input-len 550 --sonnet-output-len 150
78
79	# HuggingFace dataset
80	--dataset-name hf --dataset-path "lmarena-ai/VisionArena-Chat" --hf-split test
81
82	# General overrides (map to dataset-specific args)
83	--input-len 512 --output-len 256
84	```
85
86	## Load Control
87
88	```bash
89	# Fixed request rate (Poisson process)
90	--request-rate 10
91
92	# More bursty arrivals (gamma distribution, burstiness < 1)
93	--request-rate 10 --burstiness 0.5
94
95	# Ramp-up from low to high RPS
96	--ramp-up-strategy linear --ramp-up-start-rps 1 --ramp-up-end-rps 50
97
98	# Limit concurrency (useful for rate-limited APIs)
99	--max-concurrency 32
100	```
101
102	## Results and Metrics
103
104	\| Argument \| Description \|
105	\|----------\|-------------\|
106	\| `--save-result` \| Save benchmark results to JSON \|
107	\| `--save-detailed` \| Include per-request TTFT, TPOT, errors in JSON \|
108	\| `--append-result` \| Append to existing result file \|
109	\| `--result-dir` \| Directory for result files \|
110	\| `--result-filename` \| Custom filename (default: `{label}-{request_rate}qps-{model}-{timestamp}.json`) \|
111	\| `--percentile-metrics` \| Metrics for percentiles: `ttft`, `tpot`, `itl`, `e2el` (default: `ttft,tpot,itl`) \|
112	\| `--metric-percentiles` \| Percentile values, e.g. `25,50,99` (default: `99`) \|
113	\| `--goodput` \| SLO for goodput: `ttft:500 tpot:50` (ms) \|
114
115	## Sampling Parameters (OpenAI-compatible backends)
116
117	```bash
118	--temperature 0.7 --top-p 0.95 --top-k 50
119	--frequency-penalty 0 --presence-penalty 0 --repetition-penalty 1.0
120	```
121
122	## Common Workflows
123
124	1. Throughput test with random dataset (burst):
125
126	```bash
127	vllm bench serve --backend openai-chat --host 127.0.0.1 --port 8000 \
128	--model Qwen/Qwen2.5-1.5B-Instruct \
129	--endpoint /v1/chat/completions \
130	--dataset-name random \
131	--num-prompts 500 --random-input-len 512 --random-output-len 128
132	```
133
134	2. Latency test with fixed QPS:
135
136	```bash
137	vllm bench serve --backend openai-chat --host 127.0.0.1 --port

People who liked this also viewed

6 picks

Type