vllm-prefix-cache-bench

Name: vllm-prefix-cache-bench
Author: vllm-project

89stars23forks

How to install vllm-prefix-cache-bench?

vllm-project/vllm-skills/vllm-prefix-cache-bench

$npx -y skills add vllm-project/vllm-skills --skill vllm-prefix-cache-bench

Installs into the current project.

Summary

This is a skill for benchmarking the efficiency of automatic prefix caching in vLLM using fixed prompts, real-world datasets, or synthetic prefix/suffix patterns. Use when the user asks to benchmark prefix caching hit rate, caching efficiency, or repeated-prompt performance in vL

Files · 1

View on GitHub

SKILL.md

1	# vLLM Prefix Caching Benchmark
2
3	Benchmark the efficiency of vLLM's automatic prefix caching (APC) feature. The offline script `benchmarks/benchmark_prefix_caching.py` runs directly against the vLLM engine (no server required). For online/serving tests, use `vllm bench serve` with the `prefix_repetition` dataset.
4
5	## When to use
6
7	- User wants to measure the performance impact of prefix caching for repeated or partially-shared prompts.
8	- User wants to compare throughput/latency with and without `--enable-prefix-caching`.
9	- User wants to test prefix caching using a fixed synthetic prompt, a real dataset (e.g. ShareGPT), or a synthetic prefix/suffix repetition pattern.
10
11	## Option 1 (default). Fixed Prompt with Prefix Caching
12
13	Runs a synthetic benchmark with a fixed prompt repeated multiple times to directly measure cache hit efficiency. No dataset download required.
14
15	```bash
16	python3 benchmarks/benchmark_prefix_caching.py \
17	--model Qwen/Qwen3-8B \
18	--enable-prefix-caching \
19	--num-prompts 1 \
20	--repeat-count 100 \
21	--input-length-range 128:256
22	```
23
24	To compare against the baseline without caching:
25
26	```bash
27	python3 benchmarks/benchmark_prefix_caching.py \
28	--model Qwen/Qwen3-8B \
29	--no-enable-prefix-caching \
30	--num-prompts 1 \
31	--repeat-count 100 \
32	--input-length-range 128:256
33	```
34
35	## Option 2. ShareGPT Dataset with Prefix Caching
36
37	Uses real-world conversational data from ShareGPT to evaluate prefix caching with naturally occurring prompt sharing.
38
39	First, download the dataset:
40
41	```bash
42	wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
43	```
44
45	Then run the benchmark:
46
47	```bash
48	python3 benchmarks/benchmark_prefix_caching.py \
49	--model Qwen/Qwen3-8B \
50	--dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
51	--enable-prefix-caching \
52	--num-prompts 20 \
53	--repeat-count 5 \
54	--input-length-range 128:256
55	```
56
57	## Option 3. Prefix Repetition Dataset (Online)
58
59	Uses `vllm bench serve` with the synthetic `prefix_repetition` dataset to test caching via the serving API. This requires a running vLLM server.
60
61	First, start the server:
62
63	```bash
64	vllm serve Qwen/Qwen3-8B
65	```
66
67	Then run the benchmark:
68
69	```bash
70	vllm bench serve \
71	--backend openai \
72	--model Qwen/Qwen3-8B \
73	--dataset-name prefix_repetition \
74	--num-prompts 100 \
75	--prefix-repetition-prefix-len 512 \
76	--prefix-repetition-suffix-len 128 \
77	--prefix-repetition-num-prefixes 5 \
78	--prefix-repetition-output-len 128
79	```
80
81	Key parameters for `prefix_repetition`:
82
83	\| Parameter \| Description \|
84	\|---\|---\|
85	\| `--prefix-repetition-prefix-len` \| Number of tokens in the shared prefix portion \|
86	\| `--prefix-repetition-suffix-len` \| Number of tokens in the unique suffix portion \|
87	\| `--prefix-repetition-num-prefixes` \| Number of distinct prefixes to cycle through \|
88	\| `--prefix-repetition-output-len` \| Number of output tokens to generate per request \|
89
90	## Notes
91
92	- Run all commands from the root of the vLLM repository (`cd vllm`).
93	- Keep the default model (`Qwen/Qwen3-8B`) unless the user specifies a different one or the model is unavailable; change only `--model`.
94	- `--repeat-count` in Option 1 and 2 controls how many times each sampled prompt is replayed; higher values increase cache hit rate.
95	- `--input-length-range` accepts a `min:max` token range, e.g. `128:256`.
96	- For multi-GPU setups, add `--tensor-parallel-size <N>`.
97	- To test different hash algorithms for prefix caching internals, use `--prefix-caching-hash-algo xxhash` (requires `pip install xxhash`).
98
99	## Arguments for `benchmark_prefix_caching.py`
100
101	\| Argument \| Required \| Description \|
102	\|---\|---\|---\|
103	\| `--model` \| Yes \| Model name or path (HuggingFace ID or local path) \|
104	\| `--num-prompts` \| Yes \| Number of prompts to process \|
105	\| `--input-length-range` \| Yes \| Token length range for inputs, e.g. `128:256` \|
106	\| `--repeat-count` \| No \| Number of times each prompt is repeated (default: 1) \|
107	\| `--dataset-path` \| No \| Path to a dataset file (e.g. ShareGPT JSON). Omit for synthetic fixed-prompt mode \|
108	\| `--prefix-len` \| No \| Fixed prefix token length to prepend to every prompt \|
109	\| `--output-len` \| No \| Number of output tokens to generate per request \|
110	\| `--sort` \| No \| Sort prompts by length before benchmarking \|
111	\| `--enable-prefix-caching` / `--no-enable-prefix-caching` \| No \| Toggle APC (recommended: enable to test caching) \|
112	\| `--prefix-caching-hash-algo` \| No \| Hash algorithm: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` \|
113	\| `--tensor-parallel-size` \| No \| Number of GPUs for tensor parallelism \|
114	\| `--disable-detokenize` \| No \| Skip detokenization to reduce overhead \|
115
116	## Troubleshooting
117
118	- If

People who liked this also viewed

6 picks

Type

vllm-prefix-cache-bench

89stars23forks

View on GitHub

How to install vllm-prefix-cache-bench?

vllm-project/vllm-skills/vllm-prefix-cache-bench

$npx -y skills add vllm-project/vllm-skills --skill vllm-prefix-cache-bench

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# vLLM Prefix Caching Benchmark
2
3	Benchmark the efficiency of vLLM's automatic prefix caching (APC) feature. The offline script `benchmarks/benchmark_prefix_caching.py` runs directly against the vLLM engine (no server required). For online/serving tests, use `vllm bench serve` with the `prefix_repetition` dataset.
4
5	## When to use
6
7	- User wants to measure the performance impact of prefix caching for repeated or partially-shared prompts.
8	- User wants to compare throughput/latency with and without `--enable-prefix-caching`.
9	- User wants to test prefix caching using a fixed synthetic prompt, a real dataset (e.g. ShareGPT), or a synthetic prefix/suffix repetition pattern.
10
11	## Option 1 (default). Fixed Prompt with Prefix Caching
12
13	Runs a synthetic benchmark with a fixed prompt repeated multiple times to directly measure cache hit efficiency. No dataset download required.
14
15	```bash
16	python3 benchmarks/benchmark_prefix_caching.py \
17	--model Qwen/Qwen3-8B \
18	--enable-prefix-caching \
19	--num-prompts 1 \
20	--repeat-count 100 \
21	--input-length-range 128:256
22	```
23
24	To compare against the baseline without caching:
25
26	```bash
27	python3 benchmarks/benchmark_prefix_caching.py \
28	--model Qwen/Qwen3-8B \
29	--no-enable-prefix-caching \
30	--num-prompts 1 \
31	--repeat-count 100 \
32	--input-length-range 128:256
33	```
34
35	## Option 2. ShareGPT Dataset with Prefix Caching
36
37	Uses real-world conversational data from ShareGPT to evaluate prefix caching with naturally occurring prompt sharing.
38
39	First, download the dataset:
40
41	```bash
42	wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
43	```
44
45	Then run the benchmark:
46
47	```bash
48	python3 benchmarks/benchmark_prefix_caching.py \
49	--model Qwen/Qwen3-8B \
50	--dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
51	--enable-prefix-caching \
52	--num-prompts 20 \
53	--repeat-count 5 \
54	--input-length-range 128:256
55	```
56
57	## Option 3. Prefix Repetition Dataset (Online)
58
59	Uses `vllm bench serve` with the synthetic `prefix_repetition` dataset to test caching via the serving API. This requires a running vLLM server.
60
61	First, start the server:
62
63	```bash
64	vllm serve Qwen/Qwen3-8B
65	```
66
67	Then run the benchmark:
68
69	```bash
70	vllm bench serve \
71	--backend openai \
72	--model Qwen/Qwen3-8B \
73	--dataset-name prefix_repetition \
74	--num-prompts 100 \
75	--prefix-repetition-prefix-len 512 \
76	--prefix-repetition-suffix-len 128 \
77	--prefix-repetition-num-prefixes 5 \
78	--prefix-repetition-output-len 128
79	```
80
81	Key parameters for `prefix_repetition`:
82
83	\| Parameter \| Description \|
84	\|---\|---\|
85	\| `--prefix-repetition-prefix-len` \| Number of tokens in the shared prefix portion \|
86	\| `--prefix-repetition-suffix-len` \| Number of tokens in the unique suffix portion \|
87	\| `--prefix-repetition-num-prefixes` \| Number of distinct prefixes to cycle through \|
88	\| `--prefix-repetition-output-len` \| Number of output tokens to generate per request \|
89
90	## Notes
91
92	- Run all commands from the root of the vLLM repository (`cd vllm`).
93	- Keep the default model (`Qwen/Qwen3-8B`) unless the user specifies a different one or the model is unavailable; change only `--model`.
94	- `--repeat-count` in Option 1 and 2 controls how many times each sampled prompt is replayed; higher values increase cache hit rate.
95	- `--input-length-range` accepts a `min:max` token range, e.g. `128:256`.
96	- For multi-GPU setups, add `--tensor-parallel-size <N>`.
97	- To test different hash algorithms for prefix caching internals, use `--prefix-caching-hash-algo xxhash` (requires `pip install xxhash`).
98
99	## Arguments for `benchmark_prefix_caching.py`
100
101	\| Argument \| Required \| Description \|
102	\|---\|---\|---\|
103	\| `--model` \| Yes \| Model name or path (HuggingFace ID or local path) \|
104	\| `--num-prompts` \| Yes \| Number of prompts to process \|
105	\| `--input-length-range` \| Yes \| Token length range for inputs, e.g. `128:256` \|
106	\| `--repeat-count` \| No \| Number of times each prompt is repeated (default: 1) \|
107	\| `--dataset-path` \| No \| Path to a dataset file (e.g. ShareGPT JSON). Omit for synthetic fixed-prompt mode \|
108	\| `--prefix-len` \| No \| Fixed prefix token length to prepend to every prompt \|
109	\| `--output-len` \| No \| Number of output tokens to generate per request \|
110	\| `--sort` \| No \| Sort prompts by length before benchmarking \|
111	\| `--enable-prefix-caching` / `--no-enable-prefix-caching` \| No \| Toggle APC (recommended: enable to test caching) \|
112	\| `--prefix-caching-hash-algo` \| No \| Hash algorithm: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` \|
113	\| `--tensor-parallel-size` \| No \| Number of GPUs for tensor parallelism \|
114	\| `--disable-detokenize` \| No \| Skip detokenization to reduce overhead \|
115
116	## Troubleshooting
117
118	- If

People who liked this also viewed

6 picks

Type