vllm-deploy-k8s

Name: vllm-deploy-k8s
Author: vllm-project

89stars23forks

How to install vllm-deploy-k8s?

vllm-project/vllm-skills/vllm-deploy-k8s

$npx -y skills add vllm-project/vllm-skills --skill vllm-deploy-k8s

Installs into the current project.

Summary

Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint. Use this skill whenever the user wants to deploy, run, or serve vLLM on a Kubernetes cluster, including creating deployments, services, checking existing deployments, or managing

Files · 1

View on GitHub

SKILL.md

1	# vLLM Kubernetes Deployment
2
3	A Claude skill for deploying vLLM to Kubernetes using YAML templates. Deploys a vLLM OpenAI-compatible server as a Kubernetes Deployment with a ClusterIP Service, GPU resources, and health probes.
4
5	## What this skill does
6
7	- Deploy vLLM as a Kubernetes Deployment + Service with NVIDIA GPU support
8	- Check if a vLLM deployment already exists before deploying
9	- Check if the Hugging Face token secret exists, and ask the user for their token if not
10	- Use the `vllm/vllm-openai:latest` image by default (user can specify a different version)
11	- Provide sensible default configuration that users can customize (model, replicas, GPU count, extra vLLM flags, etc.)
12
13	## Prerequisites
14
15	- `kubectl` configured with access to a Kubernetes cluster
16	- NVIDIA GPU Operator or device plugin installed on cluster nodes
17	- Hugging Face token (required for gated models like Llama, optional for public models)
18
19	## Deployment Steps
20
21	### Step 1: Check HF token secret
22
23	Before deploying, check if the `hf-token` Kubernetes secret exists in the target namespace:
24
25	```bash
26	kubectl get secret hf-token -n <namespace>
27	```
28
29	- If the secret exists: proceed to Step 2.
30	- If the secret does not exist: ask the user to provide their Hugging Face token, then create the secret:
31
32	```bash
33	kubectl create secret generic hf-token --from-literal=HF_TOKEN="<user-provided-token>" -n <namespace>
34	```
35
36	This is required for gated models (e.g., `meta-llama/Meta-Llama-3.1-8B`). For public models, the secret is optional but recommended to avoid rate limits.
37
38	### Step 2: Check if deployment already exists
39
40	Before applying, check if a vLLM deployment already exists:
41
42	```bash
43	kubectl get deployment vllm -n <namespace>
44	```
45
46	- If it exists: inform the user that the deployment already exists. Show the current image and status. Ask the user if they want to update it or skip.
47	- If it does not exist: proceed to deploy.
48
49	### Step 3: Deploy
50
51	Apply the template YAML files to deploy vLLM:
52
53	```bash
54	kubectl apply -f templates/vllm-service.yaml -n <namespace>
55	kubectl apply -f templates/vllm-deployment.yaml -n <namespace>
56	```
57
58	### Step 4: Wait and verify
59
60	Wait for the deployment to roll out:
61
62	```bash
63	kubectl rollout status deployment/vllm -n <namespace> --timeout=600s
64	```
65
66	Verify the pod is running and ready:
67
68	```bash
69	kubectl get pods -n <namespace> -l app=vllm
70	```
71
72	Confirm the pod shows `READY 1/1` and `STATUS Running`. If the pod is not ready yet, wait and check again. If it's in `CrashLoopBackOff` or `Error`, check the logs with `kubectl logs -n <namespace> -l app=vllm`.
73
74	### Step 5: Print deployment summary
75
76	Once the pod is ready, print a summary message to the user in this format (replace placeholders with actual values):
77
78	```
79	🎉 vLLM Deployment Successful!
80
81	\| Resource \| Name \| Status \|
82	\|----------\|------\|--------\|
83	\| Deployment \| <deployment-name> \| <ready>/<total> Ready \|
84	\| Service \| <service-name> \| ClusterIP:<port> \|
85	\| Pod \| <pod-name> \| Running \|
86	\| Image \| <image> \| \|
87	\| Model \| <model> \| \|
88
89
90
91	To test the API, run these two commands in your terminal:
92
93	1. Open a port-forward (this connects your local port <port> to the vLLM service inside the cluster):
94
95	kubectl port-forward svc/vllm-svc <port>:<port> -n <namespace>
96
97	2. In a separate terminal, send a test request to the OpenAI-compatible API:
98
99	curl -s http://localhost:<port>/v1/chat/completions \
100	-H "Content-Type: application/json" \
101	-d '{"model":"<model>","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}' \| python3 -m json.tool
102
103	If everything is working, you'll get a JSON response with the model's reply.
104	```
105
106	## Default Configuration
107
108	The templates use the following defaults:
109
110	\| Parameter \| Default Value \|
111	\|-----------\|---------------\|
112	\| Image \| `vllm/vllm-openai:latest` \|
113	\| Model \| `Qwen/Qwen2.5-1.5B-Instruct` \|
114	\| Port \| `8000` \|
115	\| Replicas \| `1` \|
116	\| GPU count \| `1` \|
117	\| GPU memory utilization \| `0.85` \|
118	\| Tensor parallel size \| `1` \|
119	\| CPU request / limit \| `12` / `128` \|
120	\| Memory request / limit \| `100Gi` / `400Gi` \|
121	\| Shared memory (dshm) \| `80Gi` \|
122
123	## Customization
124
125	When the user requests changes, modify the template YAML files before applying. The following can be customized:
126
127	- Image version: Change `image: vllm/vllm-openai:<version>` in `templates/vllm-deployment.yaml` (default: `latest`). Use a specific version tag like `v0.17.1` if the user requests it.
128	- Model: Change the model name in the `vllm serve` command inside the Deployment `args`.
129	- Extra vLLM flags: Append additional flags to the `vllm serve` command in the Deployment `args` (e.g., `--max-model-len 4096`, `--kv-cache-dtype fp8`, `--enf

People who liked this also viewed

6 picks

Type

vllm-deploy-k8s

89stars23forks

View on GitHub

How to install vllm-deploy-k8s?

vllm-project/vllm-skills/vllm-deploy-k8s

$npx -y skills add vllm-project/vllm-skills --skill vllm-deploy-k8s

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# vLLM Kubernetes Deployment
2
3	A Claude skill for deploying vLLM to Kubernetes using YAML templates. Deploys a vLLM OpenAI-compatible server as a Kubernetes Deployment with a ClusterIP Service, GPU resources, and health probes.
4
5	## What this skill does
6
7	- Deploy vLLM as a Kubernetes Deployment + Service with NVIDIA GPU support
8	- Check if a vLLM deployment already exists before deploying
9	- Check if the Hugging Face token secret exists, and ask the user for their token if not
10	- Use the `vllm/vllm-openai:latest` image by default (user can specify a different version)
11	- Provide sensible default configuration that users can customize (model, replicas, GPU count, extra vLLM flags, etc.)
12
13	## Prerequisites
14
15	- `kubectl` configured with access to a Kubernetes cluster
16	- NVIDIA GPU Operator or device plugin installed on cluster nodes
17	- Hugging Face token (required for gated models like Llama, optional for public models)
18
19	## Deployment Steps
20
21	### Step 1: Check HF token secret
22
23	Before deploying, check if the `hf-token` Kubernetes secret exists in the target namespace:
24
25	```bash
26	kubectl get secret hf-token -n <namespace>
27	```
28
29	- If the secret exists: proceed to Step 2.
30	- If the secret does not exist: ask the user to provide their Hugging Face token, then create the secret:
31
32	```bash
33	kubectl create secret generic hf-token --from-literal=HF_TOKEN="<user-provided-token>" -n <namespace>
34	```
35
36	This is required for gated models (e.g., `meta-llama/Meta-Llama-3.1-8B`). For public models, the secret is optional but recommended to avoid rate limits.
37
38	### Step 2: Check if deployment already exists
39
40	Before applying, check if a vLLM deployment already exists:
41
42	```bash
43	kubectl get deployment vllm -n <namespace>
44	```
45
46	- If it exists: inform the user that the deployment already exists. Show the current image and status. Ask the user if they want to update it or skip.
47	- If it does not exist: proceed to deploy.
48
49	### Step 3: Deploy
50
51	Apply the template YAML files to deploy vLLM:
52
53	```bash
54	kubectl apply -f templates/vllm-service.yaml -n <namespace>
55	kubectl apply -f templates/vllm-deployment.yaml -n <namespace>
56	```
57
58	### Step 4: Wait and verify
59
60	Wait for the deployment to roll out:
61
62	```bash
63	kubectl rollout status deployment/vllm -n <namespace> --timeout=600s
64	```
65
66	Verify the pod is running and ready:
67
68	```bash
69	kubectl get pods -n <namespace> -l app=vllm
70	```
71
72	Confirm the pod shows `READY 1/1` and `STATUS Running`. If the pod is not ready yet, wait and check again. If it's in `CrashLoopBackOff` or `Error`, check the logs with `kubectl logs -n <namespace> -l app=vllm`.
73
74	### Step 5: Print deployment summary
75
76	Once the pod is ready, print a summary message to the user in this format (replace placeholders with actual values):
77
78	```
79	🎉 vLLM Deployment Successful!
80
81	\| Resource \| Name \| Status \|
82	\|----------\|------\|--------\|
83	\| Deployment \| <deployment-name> \| <ready>/<total> Ready \|
84	\| Service \| <service-name> \| ClusterIP:<port> \|
85	\| Pod \| <pod-name> \| Running \|
86	\| Image \| <image> \| \|
87	\| Model \| <model> \| \|
88
89
90
91	To test the API, run these two commands in your terminal:
92
93	1. Open a port-forward (this connects your local port <port> to the vLLM service inside the cluster):
94
95	kubectl port-forward svc/vllm-svc <port>:<port> -n <namespace>
96
97	2. In a separate terminal, send a test request to the OpenAI-compatible API:
98
99	curl -s http://localhost:<port>/v1/chat/completions \
100	-H "Content-Type: application/json" \
101	-d '{"model":"<model>","messages":[{"role":"user","content":"Hello!"}],"max_tokens":50}' \| python3 -m json.tool
102
103	If everything is working, you'll get a JSON response with the model's reply.
104	```
105
106	## Default Configuration
107
108	The templates use the following defaults:
109
110	\| Parameter \| Default Value \|
111	\|-----------\|---------------\|
112	\| Image \| `vllm/vllm-openai:latest` \|
113	\| Model \| `Qwen/Qwen2.5-1.5B-Instruct` \|
114	\| Port \| `8000` \|
115	\| Replicas \| `1` \|
116	\| GPU count \| `1` \|
117	\| GPU memory utilization \| `0.85` \|
118	\| Tensor parallel size \| `1` \|
119	\| CPU request / limit \| `12` / `128` \|
120	\| Memory request / limit \| `100Gi` / `400Gi` \|
121	\| Shared memory (dshm) \| `80Gi` \|
122
123	## Customization
124
125	When the user requests changes, modify the template YAML files before applying. The following can be customized:
126
127	- Image version: Change `image: vllm/vllm-openai:<version>` in `templates/vllm-deployment.yaml` (default: `latest`). Use a specific version tag like `v0.17.1` if the user requests it.
128	- Model: Change the model name in the `vllm serve` command inside the Deployment `args`.
129	- Extra vLLM flags: Append additional flags to the `vllm serve` command in the Deployment `args` (e.g., `--max-model-len 4096`, `--kv-cache-dtype fp8`, `--enf

People who liked this also viewed

6 picks

Type