vllm-deploy-docker

Name: vllm-deploy-docker
Author: vllm-project

89stars23forks

How to install vllm-deploy-docker?

vllm-project/vllm-skills/vllm-deploy-docker

$npx -y skills add vllm-project/vllm-skills --skill vllm-deploy-docker

Installs into the current project.

Summary

Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.

Files · 1

View on GitHub

SKILL.md

1	# vLLM Docker Deployment
2
3	A Claude skill describing how to deploy vLLM with Docker using the official pre-built images or building the image from source supporting NVIDIA GPUs with CUDA. Instructions include NVIDIA CUDA support, example `docker run` and a minimal `docker-compose` snippet, recommended flags, and troubleshooting notes. For AMD, Intel, or other accelerators, please refer to the [vLLM documentation](https://docs.vllm.ai/) for alternative deployment methods.
4
5	## What this skill does
6
7	- Deploy vLLM with docker using pre-built images (recommended for most users) or build from source for custom configurations
8	- Provide example commands for running the OpenAI-compatible server with GPU access and mounted Hugging Face cache
9	- Point to build-from-source instructions when a custom image or optional dependencies are needed
10	- Explain common flags: `--ipc=host`, shared cache mounts, and `HF_TOKEN` handling
11
12	## Prerequisites
13
14	- Docker Engine installed (Docker 20.10+ recommended)
15	- NVIDIA GPU(s) with appropriate drivers and CUDA toolkit installed
16	- Optional: `curl` for API tests
17	- A Hugging Face token if pulling private models or to avoid rate-limits: `HF_TOKEN`
18
19	## Quickstart using Pre-built Image (recommended)
20
21	Run a vLLM OpenAI-compatible server with GPU access, mounting the HF cache and forwarding port 8000:
22
23	```bash
24	docker run --rm --gpus all \
25	-v ~/.cache/huggingface:/root/.cache/huggingface \
26	--env "HF_TOKEN=$HF_TOKEN" \
27	-p 8000:8000 \
28	--ipc=host \
29	vllm/vllm-openai:latest \
30	--model Qwen/Qwen2.5-1.5B-Instruct
31	```
32
33	- `--gpus all` exposes all GPUs to the container. Adjust if you need specific GPUs.
34	- `--ipc=host` or an appropriately large `--shm-size` is recommended so PyTorch and vLLM can share host shared memory.
35	- Mounting `~/.cache/huggingface` avoids re-downloading models inside the container.
36
37	> Note: vLLM and this skill recommend using the latest Docker image (`vllm/vllm-openai:latest`). For legacy version images, you may refer to the [Docker Hub image tags](https://hub.docker.com/r/vllm/vllm-openai/tags).
38
39	## Build Docker image from source
40
41	You can build and run vLLM from source by using the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile).
42	First, check the hardware of the host machine and ensure you have the necessary dependencies installed (e.g., NVIDIA drivers, CUDA toolkit, Docker with BuildKit support). For ARM64/aarch64 builds, refer to the "Building for ARM64/aarch64" section.
43
44	### Basic build command
45
46	```bash
47	DOCKER_BUILDKIT=1 docker build . \
48	--target vllm-openai \
49	--tag vllm/vllm-openai \
50	--file docker/Dockerfile
51	```
52
53	The `--target vllm-openai` specifies that you are building the OpenAI-compatible server image. The `DOCKER_BUILDKIT=1` environment variable enables BuildKit, which provides better caching and faster builds.
54
55	### Build arguments and options
56
57	- `--build-arg max_jobs=<N>` — sets the number of parallel compilation jobs for building CUDA kernels. Useful for speeding up builds on multi-core systems.
58	- `--build-arg nvcc_threads=<N>` — controls CUDA compiler threads. Recommended to use a smaller value than `max_jobs` to avoid excessive memory usage.
59	- `--build-arg torch_cuda_arch_list=""` — if set to empty string, vLLM will detect and build only for the current GPU's compute capability. By default, vLLM builds for all GPU types for wider distribution.
60
61	### Using precompiled wheels to speed up builds
62
63	If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time:
64
65	- Enable precompiled wheels: Add `--build-arg VLLM_USE_PRECOMPILED="1"` to your build command.
66	- How it works: By default, vLLM automatically finds the correct precompiled wheels from the [Nightly Builds](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/) by using the merge-base commit with the upstream `main` branch.
67	- Specify a commit: To use wheels from a specific commit, add `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>`.
68
69	Example with precompiled wheels and options for fast compilation:
70	```bash
71	DOCKER_BUILDKIT=1 docker build . \
72	--target vllm-openai \
73	--tag vllm/vllm-openai \
74	--file docker/Dockerfile \
75	--build-arg max_jobs=8 \
76	--build-arg nvcc_threads=2 \
77	--build-arg VLLM_USE_PRECOMPILED="1"
78	```
79
80	### Building with optional dependencies (optional)
81
82	vLLM does not include optional dependencies (e.g., audio processing) in the pre-built image to avoid licensing issues. If you need optional dependencies, create a custom Dockerfile that extends the base image:
83
84	Example: adding audio optional dependencies
85	```dockerfile
86	# NOTE: MAKE SURE the version of vLLM matches the base image!
87	FROM vllm/vllm-openai:0.11.0
88
89	# Install audio optional dependencies
90	RUN uv pi

People who liked this also viewed

6 picks

Type

microsoft-foundryDeploy, evaluate, fine-tune, and manage Foundry agents end-to-end with azd: hosted agent scaffold/run/deploy, prompt agent create, batch eval, continuous eval, prompt optimizer, Agent Optimizer scaffoSkillsJul 2026476k1.3k
azure-deployExecute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application SkillsJul 2026472k1.3k
azure-preparePrepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wSkillsJul 2026471k1.3k
azure-validate--- name: azure-validate description: "Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity pSkillsJul 2026471k1.3k
azure-aigatewayConfigure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: semantic caching, token limit, content safety, load balancing, AI model governance, MCP rate limiting, jailbSkillsJul 2026470k1.3k
azure-hosted-copilot-sdkOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026424k1.3k

vllm-deploy-docker

89stars23forks

View on GitHub

How to install vllm-deploy-docker?

vllm-project/vllm-skills/vllm-deploy-docker

$npx -y skills add vllm-project/vllm-skills --skill vllm-deploy-docker

Installs into the current project.

Summary

Deploy vLLM using Docker (pre-built images or build-from-source) with NVIDIA GPU support and run the OpenAI-compatible server.

Files · 1

View on GitHub

SKILL.md

1	# vLLM Docker Deployment
2
3	A Claude skill describing how to deploy vLLM with Docker using the official pre-built images or building the image from source supporting NVIDIA GPUs with CUDA. Instructions include NVIDIA CUDA support, example `docker run` and a minimal `docker-compose` snippet, recommended flags, and troubleshooting notes. For AMD, Intel, or other accelerators, please refer to the [vLLM documentation](https://docs.vllm.ai/) for alternative deployment methods.
4
5	## What this skill does
6
7	- Deploy vLLM with docker using pre-built images (recommended for most users) or build from source for custom configurations
8	- Provide example commands for running the OpenAI-compatible server with GPU access and mounted Hugging Face cache
9	- Point to build-from-source instructions when a custom image or optional dependencies are needed
10	- Explain common flags: `--ipc=host`, shared cache mounts, and `HF_TOKEN` handling
11
12	## Prerequisites
13
14	- Docker Engine installed (Docker 20.10+ recommended)
15	- NVIDIA GPU(s) with appropriate drivers and CUDA toolkit installed
16	- Optional: `curl` for API tests
17	- A Hugging Face token if pulling private models or to avoid rate-limits: `HF_TOKEN`
18
19	## Quickstart using Pre-built Image (recommended)
20
21	Run a vLLM OpenAI-compatible server with GPU access, mounting the HF cache and forwarding port 8000:
22
23	```bash
24	docker run --rm --gpus all \
25	-v ~/.cache/huggingface:/root/.cache/huggingface \
26	--env "HF_TOKEN=$HF_TOKEN" \
27	-p 8000:8000 \
28	--ipc=host \
29	vllm/vllm-openai:latest \
30	--model Qwen/Qwen2.5-1.5B-Instruct
31	```
32
33	- `--gpus all` exposes all GPUs to the container. Adjust if you need specific GPUs.
34	- `--ipc=host` or an appropriately large `--shm-size` is recommended so PyTorch and vLLM can share host shared memory.
35	- Mounting `~/.cache/huggingface` avoids re-downloading models inside the container.
36
37	> Note: vLLM and this skill recommend using the latest Docker image (`vllm/vllm-openai:latest`). For legacy version images, you may refer to the [Docker Hub image tags](https://hub.docker.com/r/vllm/vllm-openai/tags).
38
39	## Build Docker image from source
40
41	You can build and run vLLM from source by using the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile).
42	First, check the hardware of the host machine and ensure you have the necessary dependencies installed (e.g., NVIDIA drivers, CUDA toolkit, Docker with BuildKit support). For ARM64/aarch64 builds, refer to the "Building for ARM64/aarch64" section.
43
44	### Basic build command
45
46	```bash
47	DOCKER_BUILDKIT=1 docker build . \
48	--target vllm-openai \
49	--tag vllm/vllm-openai \
50	--file docker/Dockerfile
51	```
52
53	The `--target vllm-openai` specifies that you are building the OpenAI-compatible server image. The `DOCKER_BUILDKIT=1` environment variable enables BuildKit, which provides better caching and faster builds.
54
55	### Build arguments and options
56
57	- `--build-arg max_jobs=<N>` — sets the number of parallel compilation jobs for building CUDA kernels. Useful for speeding up builds on multi-core systems.
58	- `--build-arg nvcc_threads=<N>` — controls CUDA compiler threads. Recommended to use a smaller value than `max_jobs` to avoid excessive memory usage.
59	- `--build-arg torch_cuda_arch_list=""` — if set to empty string, vLLM will detect and build only for the current GPU's compute capability. By default, vLLM builds for all GPU types for wider distribution.
60
61	### Using precompiled wheels to speed up builds
62
63	If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time:
64
65	- Enable precompiled wheels: Add `--build-arg VLLM_USE_PRECOMPILED="1"` to your build command.
66	- How it works: By default, vLLM automatically finds the correct precompiled wheels from the [Nightly Builds](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/) by using the merge-base commit with the upstream `main` branch.
67	- Specify a commit: To use wheels from a specific commit, add `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=<commit_hash>`.
68
69	Example with precompiled wheels and options for fast compilation:
70	```bash
71	DOCKER_BUILDKIT=1 docker build . \
72	--target vllm-openai \
73	--tag vllm/vllm-openai \
74	--file docker/Dockerfile \
75	--build-arg max_jobs=8 \
76	--build-arg nvcc_threads=2 \
77	--build-arg VLLM_USE_PRECOMPILED="1"
78	```
79
80	### Building with optional dependencies (optional)
81
82	vLLM does not include optional dependencies (e.g., audio processing) in the pre-built image to avoid licensing issues. If you need optional dependencies, create a custom Dockerfile that extends the base image:
83
84	Example: adding audio optional dependencies
85	```dockerfile
86	# NOTE: MAKE SURE the version of vLLM matches the base image!
87	FROM vllm/vllm-openai:0.11.0
88
89	# Install audio optional dependencies
90	RUN uv pi

People who liked this also viewed

6 picks

Type