vllm-bench-random-synthetic

Name: vllm-bench-random-synthetic
Author: vllm-project

89stars23forks

How to install vllm-bench-random-synthetic?

vllm-project/vllm-skills/vllm-bench-random-synthetic

$npx -y skills add vllm-project/vllm-skills --skill vllm-bench-random-synthetic

Installs into the current project.

Summary

Run vLLM performance benchmark using synthetic random data to measure throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and other key performance metrics. Use when the user wants to quickly test vLLM serving performance without downloading external datasets.

Files · 1

View on GitHub

SKILL.md

1	# vLLM Benchmark with Random Synthetic Data
2
3	Run a quick performance benchmark on a vLLM server using synthetic random data. This skill measures core serving metrics including request throughput, token throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and inter-token latency.
4
5	## When to use
6
7	- User wants to quickly benchmark vLLM serving performance
8	- User wants to measure throughput and latency metrics without downloading datasets
9	- User wants to test a vLLM deployment with synthetic workload
10	- User wants baseline performance numbers for a specific model
11
12	## Prerequisites
13
14	- vLLM must be installed (`pip install vllm`)
15	- A vLLM server must be running (or can be started as part of the benchmark)
16	- For GPU models, NVIDIA GPU with appropriate drivers must be available
17
18	## Quick Start
19
20	The simplest way to run the benchmark:
21
22	```bash
23	# Start vLLM server (in background or separate terminal)
24	vllm serve Qwen/Qwen2.5-1.5B-Instruct
25
26	# Run benchmark with random synthetic data
27	vllm bench serve \
28	--backend openai-chat \
29	--model Qwen/Qwen2.5-1.5B-Instruct \
30	--endpoint /v1/chat/completions \
31	--dataset-name random \
32	--num-prompts 10
33	```
34
35	Note:
36	- Use `--backend openai-chat` with endpoint `/v1/chat/completions` for online benchmarks.
37
38	## Parameters
39
40	\| Parameter \| Description \| Default \|
41	\|-----------\|-------------\|---------\|
42	\| `--backend` \| Backend type: `vllm`, `openai`, `openai-chat` \| `vllm` \|
43	\| `--model` \| Model name (must match the server) \| Required \|
44	\| `--endpoint` \| API endpoint path \| `/v1/completions` or `/v1/chat/completions` \|
45	\| `--dataset-name` \| Dataset to use \| `random` (synthetic) \|
46	\| `--num-prompts` \| Number of requests to send \| `10` \|
47	\| `--port` \| Server port \| `8000` \|
48	\| `--max-concurrency` \| Maximum concurrent requests \| Auto \|
49	\| `--save-result` \| Save results to file \| Off \|
50	\| `--result-dir` \| Directory to save results \| `./` \|
51
52	## Expected Output
53
54	When successful, you will see output like:
55
56	```
57	============ Serving Benchmark Result ============
58	Successful requests: 10
59	Benchmark duration (s): 5.78
60	Total input tokens: 1369
61	Total generated tokens: 2212
62	Request throughput (req/s): 1.73
63	Output token throughput (tok/s): 382.89
64	Total token throughput (tok/s): 619.85
65	---------------Time to First Token----------------
66	Mean TTFT (ms): 71.54
67	Median TTFT (ms): 73.88
68	P99 TTFT (ms): 79.49
69	-----Time per Output Token (excl. 1st token)------
70	Mean TPOT (ms): 7.91
71	Median TPOT (ms): 7.96
72	P99 TPOT (ms): 8.03
73	---------------Inter-token Latency----------------
74	Mean ITL (ms): 7.74
75	Median ITL (ms): 7.70
76	P99 ITL (ms): 8.39
77	==================================================
78	```
79
80	## Advanced Usage
81
82	### With more prompts for better statistics
83
84	```bash
85	vllm bench serve \
86	--backend openai-chat \
87	--model Qwen/Qwen2.5-1.5B-Instruct \
88	--endpoint /v1/chat/completions \
89	--dataset-name random \
90	--num-prompts 100
91	```
92
93	### Save results to file
94
95	```bash
96	vllm bench serve \
97	--backend openai-chat \
98	--model Qwen/Qwen2.5-1.5B-Instruct \
99	--endpoint /v1/chat/completions \
100	--dataset-name random \
101	--num-prompts 50 \
102	--save-result \
103	--result-dir ./benchmark-results/
104	```
105
106	### Custom port and concurrency
107
108	```bash
109	vllm bench serve \
110	--backend openai-chat \
111	--model meta-llama/Llama-3.1-8B-Instruct \
112	--endpoint /v1/chat/completions \
113	--dataset-name random \
114	--num-prompts 100 \
115	--port 8001 \
116	--max-concurrency 4
117	```
118
119	## Model Recommendations
120
121	For quick testing (small models, fast):
122	- `Qwen/Qwen2.5-1.5B-Instruct` (recommended for quick tests)
123	- `facebook/opt-125m`
124	- `facebook/opt-350m`
125
126	For realistic benchmarks (medium models):
127	- `Qwen/Qwen2.5-7B-Instruct`
128	- `meta-llama/Llama-3.1-8B-Instruct`
129	- `mistralai/Mistral-7B-Instruct-v0.3`
130
131	## Workflow
132
133	1. Check if vLLM is installed: Run `vllm --version` to verify
134	2. Check if server is already running: Run `curl http://localhost:8000/health` to check
135	3. Start vLLM server if needed: Run `vllm serve <model-name>` (wait for "Application startup complete")
136	4. Run benchmark: Execute `vllm bench serve` with appropriate parameters
137	5. Review results: Check throughput and latency metrics
138	6. Clean up: If the agent skill started the vLLM server (not a pre-existing one), stop it after benchmark completion using `kill <PID>`
139
140	## Troubleshooting
141
142	Server not responding:
143	- Check if

People who liked this also viewed

6 picks

Type

vllm-bench-random-synthetic

89stars23forks

View on GitHub

How to install vllm-bench-random-synthetic?

vllm-project/vllm-skills/vllm-bench-random-synthetic

$npx -y skills add vllm-project/vllm-skills --skill vllm-bench-random-synthetic

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# vLLM Benchmark with Random Synthetic Data
2
3	Run a quick performance benchmark on a vLLM server using synthetic random data. This skill measures core serving metrics including request throughput, token throughput, TTFT (Time to First Token), TPOT (Time per Output Token), and inter-token latency.
4
5	## When to use
6
7	- User wants to quickly benchmark vLLM serving performance
8	- User wants to measure throughput and latency metrics without downloading datasets
9	- User wants to test a vLLM deployment with synthetic workload
10	- User wants baseline performance numbers for a specific model
11
12	## Prerequisites
13
14	- vLLM must be installed (`pip install vllm`)
15	- A vLLM server must be running (or can be started as part of the benchmark)
16	- For GPU models, NVIDIA GPU with appropriate drivers must be available
17
18	## Quick Start
19
20	The simplest way to run the benchmark:
21
22	```bash
23	# Start vLLM server (in background or separate terminal)
24	vllm serve Qwen/Qwen2.5-1.5B-Instruct
25
26	# Run benchmark with random synthetic data
27	vllm bench serve \
28	--backend openai-chat \
29	--model Qwen/Qwen2.5-1.5B-Instruct \
30	--endpoint /v1/chat/completions \
31	--dataset-name random \
32	--num-prompts 10
33	```
34
35	Note:
36	- Use `--backend openai-chat` with endpoint `/v1/chat/completions` for online benchmarks.
37
38	## Parameters
39
40	\| Parameter \| Description \| Default \|
41	\|-----------\|-------------\|---------\|
42	\| `--backend` \| Backend type: `vllm`, `openai`, `openai-chat` \| `vllm` \|
43	\| `--model` \| Model name (must match the server) \| Required \|
44	\| `--endpoint` \| API endpoint path \| `/v1/completions` or `/v1/chat/completions` \|
45	\| `--dataset-name` \| Dataset to use \| `random` (synthetic) \|
46	\| `--num-prompts` \| Number of requests to send \| `10` \|
47	\| `--port` \| Server port \| `8000` \|
48	\| `--max-concurrency` \| Maximum concurrent requests \| Auto \|
49	\| `--save-result` \| Save results to file \| Off \|
50	\| `--result-dir` \| Directory to save results \| `./` \|
51
52	## Expected Output
53
54	When successful, you will see output like:
55
56	```
57	============ Serving Benchmark Result ============
58	Successful requests: 10
59	Benchmark duration (s): 5.78
60	Total input tokens: 1369
61	Total generated tokens: 2212
62	Request throughput (req/s): 1.73
63	Output token throughput (tok/s): 382.89
64	Total token throughput (tok/s): 619.85
65	---------------Time to First Token----------------
66	Mean TTFT (ms): 71.54
67	Median TTFT (ms): 73.88
68	P99 TTFT (ms): 79.49
69	-----Time per Output Token (excl. 1st token)------
70	Mean TPOT (ms): 7.91
71	Median TPOT (ms): 7.96
72	P99 TPOT (ms): 8.03
73	---------------Inter-token Latency----------------
74	Mean ITL (ms): 7.74
75	Median ITL (ms): 7.70
76	P99 ITL (ms): 8.39
77	==================================================
78	```
79
80	## Advanced Usage
81
82	### With more prompts for better statistics
83
84	```bash
85	vllm bench serve \
86	--backend openai-chat \
87	--model Qwen/Qwen2.5-1.5B-Instruct \
88	--endpoint /v1/chat/completions \
89	--dataset-name random \
90	--num-prompts 100
91	```
92
93	### Save results to file
94
95	```bash
96	vllm bench serve \
97	--backend openai-chat \
98	--model Qwen/Qwen2.5-1.5B-Instruct \
99	--endpoint /v1/chat/completions \
100	--dataset-name random \
101	--num-prompts 50 \
102	--save-result \
103	--result-dir ./benchmark-results/
104	```
105
106	### Custom port and concurrency
107
108	```bash
109	vllm bench serve \
110	--backend openai-chat \
111	--model meta-llama/Llama-3.1-8B-Instruct \
112	--endpoint /v1/chat/completions \
113	--dataset-name random \
114	--num-prompts 100 \
115	--port 8001 \
116	--max-concurrency 4
117	```
118
119	## Model Recommendations
120
121	For quick testing (small models, fast):
122	- `Qwen/Qwen2.5-1.5B-Instruct` (recommended for quick tests)
123	- `facebook/opt-125m`
124	- `facebook/opt-350m`
125
126	For realistic benchmarks (medium models):
127	- `Qwen/Qwen2.5-7B-Instruct`
128	- `meta-llama/Llama-3.1-8B-Instruct`
129	- `mistralai/Mistral-7B-Instruct-v0.3`
130
131	## Workflow
132
133	1. Check if vLLM is installed: Run `vllm --version` to verify
134	2. Check if server is already running: Run `curl http://localhost:8000/health` to check
135	3. Start vLLM server if needed: Run `vllm serve <model-name>` (wait for "Application startup complete")
136	4. Run benchmark: Execute `vllm bench serve` with appropriate parameters
137	5. Review results: Check throughput and latency metrics
138	6. Clean up: If the agent skill started the vLLM server (not a pre-existing one), stop it after benchmark completion using `kill <PID>`
139
140	## Troubleshooting
141
142	Server not responding:
143	- Check if

People who liked this also viewed

6 picks

Type