model-serving-kubernetes

Name: model-serving-kubernetes
Author: BagelHole

42stars4forks

How to install model-serving-kubernetes?

bagelhole/devops-security-agent-skills/model-serving-kubernetes

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill model-serving-kubernetes

Installs into the current project.

Summary

Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving.

Files · 1

View on GitHub

SKILL.md

1	# Model Serving on Kubernetes
2
3	Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
9	- Implementing canary deployments and A/B testing for ML models
10	- Autoscaling inference pods based on request rate or GPU metrics
11	- Deploying LLMs with Triton or KServe on Kubernetes
12	- Managing multiple model versions with traffic splitting
13
14	## Prerequisites
15
16	- Kubernetes 1.28+ with GPU nodes
17	- KServe installed (or Triton standalone)
18	- `kubectl` and `helm` configured
19	- NVIDIA GPU Operator installed on cluster
20
21	## KServe Installation
22
23	```bash
24	# Install KServe with Helm
25	helm repo add kserve https://kserve.github.io/helm-charts
26	helm repo update
27
28	helm install kserve kserve/kserve \
29	--namespace kserve \
30	--create-namespace \
31	--set kserve.controller.gateway.ingressGateway.className=nginx
32
33	# Verify
34	kubectl get pods -n kserve
35	kubectl get crd \| grep kserve
36	```
37
38	## Basic InferenceService (KServe)
39
40	```yaml
41	apiVersion: serving.kserve.io/v1beta1
42	kind: InferenceService
43	metadata:
44	name: sklearn-iris
45	namespace: models
46	spec:
47	predictor:
48	sklearn:
49	storageUri: gs://kfserving-examples/models/sklearn/1.0/model
50	resources:
51	requests:
52	cpu: "1"
53	memory: 2Gi
54	limits:
55	cpu: "2"
56	memory: 4Gi
57	```
58
59	```bash
60	kubectl apply -f inference-service.yaml
61
62	# Get inference service URL
63	kubectl get inferenceservice sklearn-iris -n models
64	# NAME URL READY ...
65	# sklearn-iris http://sklearn-iris.models.example.com True
66
67	# Test prediction
68	curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \
69	-H "Content-Type: application/json" \
70	-d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
71	```
72
73	## GPU-Enabled LLM InferenceService
74
75	```yaml
76	apiVersion: serving.kserve.io/v1beta1
77	kind: InferenceService
78	metadata:
79	name: llama-3-8b
80	namespace: models
81	annotations:
82	serving.kserve.io/enable-prometheus-scraping: "true"
83	spec:
84	predictor:
85	containers:
86	- name: vllm-container
87	image: vllm/vllm-openai:latest
88	args:
89	- "--model"
90	- "meta-llama/Llama-3.1-8B-Instruct"
91	- "--tensor-parallel-size"
92	- "1"
93	- "--gpu-memory-utilization"
94	- "0.90"
95	ports:
96	- containerPort: 8080
97	protocol: TCP
98	resources:
99	requests:
100	nvidia.com/gpu: "1"
101	memory: "20Gi"
102	cpu: "4"
103	limits:
104	nvidia.com/gpu: "1"
105	memory: "24Gi"
106	cpu: "8"
107	readinessProbe:
108	httpGet:
109	path: /health
110	port: 8080
111	initialDelaySeconds: 60
112	periodSeconds: 10
113	env:
114	- name: HUGGING_FACE_HUB_TOKEN
115	valueFrom:
116	secretKeyRef:
117	name: hf-token
118	key: token
119	nodeSelector:
120	nvidia.com/gpu.present: "true"
121	transformer:
122	containers:
123	- name: kserve-container
124	image: kserve/kserve-transformer:latest
125	```
126
127	## Canary Deployment (Traffic Splitting)
128
129	```yaml
130	apiVersion: serving.kserve.io/v1beta1
131	kind: InferenceService
132	metadata:
133	name: llama-3-8b
134	namespace: models
135	spec:
136	predictor:
137	canaryTrafficPercent: 20 # 20% to new version, 80% to stable
138	containers:
139	- name: vllm-container
140	image: vllm/vllm-openai:latest
141	args:
142	- "--model"
143	- "meta-llama/Llama-3.1-8B-Instruct-v2" # new model version
144	resources:
145	limits:
146	nvidia.com/gpu: "1"
147	```
148
149	```bash
150	# Gradually increase canary traffic
151	kubectl patch inferenceservice llama-3-8b -n models \
152	--type='json' \
153	-p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]'
154
155	# Promote canary to stable
156	kubectl patch inferenceservice llama-3-8b -n models \
157	--type='json' \
158	-p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]'
159	```
160
161	## Autoscaling with KEDA
162
163	```yaml
164	apiVersion: keda.sh/v1alpha1
165	kind: ScaledObject
166	metadata:
167	name: llama-scaler
168	namespace: models
169	spec:
170	scaleTargetRef:
171	apiVersion: serving.kserve.io/v1beta1
172	kind: InferenceService
173	name: llama-3-8b
174	minReplicaCount: 1
175	maxReplicaCount: 5
176	triggers:
177	- type: prometheus
178	metadata:
179	serverAddress: http://prometheus-server.monitoring:9090
180	metricName: kserve_request_count
181	threshold: "10"
182	query: \|
183	sum(rate(kserve_request_count_total{namespace="models",
184	service="llama-3-8b"}[1m]))
185	```
186
187	## NVIDIA Triton Inference Server
188
189	```yaml
190	apiVersion: apps/v1
191	kind: Deployment
192	metadata:
193	name: triton-server

People who liked this also viewed

6 picks

Type

model-serving-kubernetes

42stars4forks

View on GitHub

How to install model-serving-kubernetes?

bagelhole/devops-security-agent-skills/model-serving-kubernetes

$npx -y skills add BagelHole/DevOps-Security-Agent-Skills --skill model-serving-kubernetes

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model Serving on Kubernetes
2
3	Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
4
5	## When to Use This Skill
6
7	Use this skill when:
8	- Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
9	- Implementing canary deployments and A/B testing for ML models
10	- Autoscaling inference pods based on request rate or GPU metrics
11	- Deploying LLMs with Triton or KServe on Kubernetes
12	- Managing multiple model versions with traffic splitting
13
14	## Prerequisites
15
16	- Kubernetes 1.28+ with GPU nodes
17	- KServe installed (or Triton standalone)
18	- `kubectl` and `helm` configured
19	- NVIDIA GPU Operator installed on cluster
20
21	## KServe Installation
22
23	```bash
24	# Install KServe with Helm
25	helm repo add kserve https://kserve.github.io/helm-charts
26	helm repo update
27
28	helm install kserve kserve/kserve \
29	--namespace kserve \
30	--create-namespace \
31	--set kserve.controller.gateway.ingressGateway.className=nginx
32
33	# Verify
34	kubectl get pods -n kserve
35	kubectl get crd \| grep kserve
36	```
37
38	## Basic InferenceService (KServe)
39
40	```yaml
41	apiVersion: serving.kserve.io/v1beta1
42	kind: InferenceService
43	metadata:
44	name: sklearn-iris
45	namespace: models
46	spec:
47	predictor:
48	sklearn:
49	storageUri: gs://kfserving-examples/models/sklearn/1.0/model
50	resources:
51	requests:
52	cpu: "1"
53	memory: 2Gi
54	limits:
55	cpu: "2"
56	memory: 4Gi
57	```
58
59	```bash
60	kubectl apply -f inference-service.yaml
61
62	# Get inference service URL
63	kubectl get inferenceservice sklearn-iris -n models
64	# NAME URL READY ...
65	# sklearn-iris http://sklearn-iris.models.example.com True
66
67	# Test prediction
68	curl -X POST http://sklearn-iris.models.example.com/v1/models/sklearn-iris:predict \
69	-H "Content-Type: application/json" \
70	-d '{"instances": [[6.8, 2.8, 4.8, 1.4]]}'
71	```
72
73	## GPU-Enabled LLM InferenceService
74
75	```yaml
76	apiVersion: serving.kserve.io/v1beta1
77	kind: InferenceService
78	metadata:
79	name: llama-3-8b
80	namespace: models
81	annotations:
82	serving.kserve.io/enable-prometheus-scraping: "true"
83	spec:
84	predictor:
85	containers:
86	- name: vllm-container
87	image: vllm/vllm-openai:latest
88	args:
89	- "--model"
90	- "meta-llama/Llama-3.1-8B-Instruct"
91	- "--tensor-parallel-size"
92	- "1"
93	- "--gpu-memory-utilization"
94	- "0.90"
95	ports:
96	- containerPort: 8080
97	protocol: TCP
98	resources:
99	requests:
100	nvidia.com/gpu: "1"
101	memory: "20Gi"
102	cpu: "4"
103	limits:
104	nvidia.com/gpu: "1"
105	memory: "24Gi"
106	cpu: "8"
107	readinessProbe:
108	httpGet:
109	path: /health
110	port: 8080
111	initialDelaySeconds: 60
112	periodSeconds: 10
113	env:
114	- name: HUGGING_FACE_HUB_TOKEN
115	valueFrom:
116	secretKeyRef:
117	name: hf-token
118	key: token
119	nodeSelector:
120	nvidia.com/gpu.present: "true"
121	transformer:
122	containers:
123	- name: kserve-container
124	image: kserve/kserve-transformer:latest
125	```
126
127	## Canary Deployment (Traffic Splitting)
128
129	```yaml
130	apiVersion: serving.kserve.io/v1beta1
131	kind: InferenceService
132	metadata:
133	name: llama-3-8b
134	namespace: models
135	spec:
136	predictor:
137	canaryTrafficPercent: 20 # 20% to new version, 80% to stable
138	containers:
139	- name: vllm-container
140	image: vllm/vllm-openai:latest
141	args:
142	- "--model"
143	- "meta-llama/Llama-3.1-8B-Instruct-v2" # new model version
144	resources:
145	limits:
146	nvidia.com/gpu: "1"
147	```
148
149	```bash
150	# Gradually increase canary traffic
151	kubectl patch inferenceservice llama-3-8b -n models \
152	--type='json' \
153	-p='[{"op":"replace","path":"/spec/predictor/canaryTrafficPercent","value":50}]'
154
155	# Promote canary to stable
156	kubectl patch inferenceservice llama-3-8b -n models \
157	--type='json' \
158	-p='[{"op":"remove","path":"/spec/predictor/canaryTrafficPercent"}]'
159	```
160
161	## Autoscaling with KEDA
162
163	```yaml
164	apiVersion: keda.sh/v1alpha1
165	kind: ScaledObject
166	metadata:
167	name: llama-scaler
168	namespace: models
169	spec:
170	scaleTargetRef:
171	apiVersion: serving.kserve.io/v1beta1
172	kind: InferenceService
173	name: llama-3-8b
174	minReplicaCount: 1
175	maxReplicaCount: 5
176	triggers:
177	- type: prometheus
178	metadata:
179	serverAddress: http://prometheus-server.monitoring:9090
180	metricName: kserve_request_count
181	threshold: "10"
182	query: \|
183	sum(rate(kserve_request_count_total{namespace="models",
184	service="llama-3-8b"}[1m]))
185	```
186
187	## NVIDIA Triton Inference Server
188
189	```yaml
190	apiVersion: apps/v1
191	kind: Deployment
192	metadata:
193	name: triton-server

People who liked this also viewed

6 picks

Type