gke-inference

Name: gke-inference
Author: google

1.3kinstalls1.1kforks

How to install gke-inference?

google/skills/gke-inference

$npx -y skills add google/skills --skill gke-inference

Installs into the current project.

Summary

Deploys and optimizes AI/ML inference workloads on GKE, using GPUs, TPUs, and model servers. Use when deploying GKE inference servers, configuring GKE GPU resources for inference, or deploying LLMs on GKE. Don't use for generic batch jobs or HPC task queues (use gke-batch-hpc ins

Files · 1

View on GitHub

SKILL.md

1	# GKE AI/ML Inference
2
3	This reference covers deploying AI/ML inference workloads on GKE using Google's
4	Inference Quickstart (GIQ) and best practices for LLM serving.
5
6	> MCP Tools: `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`,
7	> `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`.
8	> CLI-only: `gcloud container ai profiles *`
9
10	## When to Use
11
12	- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
13	- Generate optimized Kubernetes manifests for inference
14	- Select GPU/TPU accelerators for model serving
15	- Configure autoscaling for LLM inference
16
17	## Prerequisites
18
19	- A golden path GKE Autopilot cluster (GPU workloads are supported via
20	ComputeClasses and NAP)
21	- `gcloud` CLI authenticated
22	- Sufficient GPU/TPU quota in the target region
23
24	## Workflow
25
26	### 1. Discovery: Find Models and Hardware
27
28	```bash
29	# List all supported models
30	gcloud container ai profiles models list --quiet
31
32	# Find valid accelerator/server combinations for a model
33	gcloud container ai profiles list --model=<MODEL_NAME> --quiet
34
35	# Example: what can run Gemma 2 9B?
36	gcloud container ai profiles list --model=gemma-2-9b-it --quiet
37	```
38
39	### 2. Generate Manifest
40
41	```bash
42	gcloud container ai profiles manifests create \
43	--model=<MODEL_NAME> \
44	--model-server=<SERVER> \
45	--accelerator-type=<ACCELERATOR> \
46	--target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
47	```
48
49	Parameters:
50
51	- `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`)
52	- `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`)
53	- `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`,
54	`nvidia-h100-80gb`)
55	- `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token
56	(optional, for latency optimization)
57
58	Example:
59
60	```bash
61	gcloud container ai profiles manifests create \
62	--model=gemma-2-9b-it \
63	--model-server=vllm \
64	--accelerator-type=nvidia-l4 \
65	--target-ntpot-milliseconds=50 --quiet > inference.yaml
66	```
67
68	### 3. Review and Deploy
69
70	```bash
71	# Review for placeholders (HF tokens, PVCs)
72	cat inference.yaml
73
74	# Deploy
75	kubectl apply -f inference.yaml
76
77	# Monitor
78	kubectl get pods -w
79	kubectl logs -f <POD_NAME>
80	```
81
82	> Some models require Hugging Face tokens. Create a Kubernetes Secret and
83	> reference it in the manifest.
84
85	## GPU ComputeClass for Inference
86
87	For Autopilot clusters, create a ComputeClass to target GPU nodes:
88
89	```yaml
90	apiVersion: cloud.google.com/v1
91	kind: ComputeClass
92	metadata:
93	name: l4-inference
94	spec:
95	priorities:
96	- machineFamily: g2
97	gpu:
98	type: nvidia-l4
99	count: 1
100	minCores: 4
101	minMemoryGb: 16
102	```
103
104	## Accelerator Selection Guide
105
106	\| Accelerator \| Best For \| Memory \| Relative Cost \|
107	\| ------------------- \| ------------------------ \| ----------- \| ------------- \|
108	\| NVIDIA T4 \| Budget inference, \| 16 GB \| Lowest \|
109	: : lightweight legacy : : :
110	: : models : : :
111	\| NVIDIA L4 (G2) \| Small-medium model \| 24 GB \| Low \|
112	: : inference, video, : : :
113	: : graphics : : :
114	\| NVIDIA RTX PRO 6000 \| Multimodal AI, \| 96 GB \| Medium \|
115	: (G4) : high-fidelity 3D, : : :
116	: : fine-tuning : : :
117	\| Cloud TPU v5e \| Cost-effective \| Varies \| Medium \|
118	: : transformer inference : : :
119	\| Cloud TPU v5p \| High-performance \| Varies \| High \|
120	: : training : : :
121	\| Cloud TPU v6e \| High-efficiency next-gen \| 32 GB/chip \| Medium-High \|
122	: (Trillium) : training & serving : : :
123	\| Cloud TPU v7x \| Ultra-scale inference & \| 192 GB/chip \| High \|
124	: (Ironwood) : agentic workflows : : :
125	\| NVIDIA A100 \| Large model inference, \| 40/80 GB \| High \|
126	: : enterprise ML : : :
127	\| NVIDIA H100 / H200 \| Frontier model training, \| 80/141 GB \| Highest \|
128	: : high throughput : : :
129	\| NVIDIA B200 (A4) \| Blackwell-scale \| 192 GB \| Highest \|
130	: : tr

People who liked this also viewed

6 picks

Type

gke-inference

1.3kinstalls1.1kforks

View on GitHub

How to install gke-inference?

google/skills/gke-inference

$npx -y skills add google/skills --skill gke-inference

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# GKE AI/ML Inference
2
3	This reference covers deploying AI/ML inference workloads on GKE using Google's
4	Inference Quickstart (GIQ) and best practices for LLM serving.
5
6	> MCP Tools: `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`,
7	> `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`.
8	> CLI-only: `gcloud container ai profiles *`
9
10	## When to Use
11
12	- Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
13	- Generate optimized Kubernetes manifests for inference
14	- Select GPU/TPU accelerators for model serving
15	- Configure autoscaling for LLM inference
16
17	## Prerequisites
18
19	- A golden path GKE Autopilot cluster (GPU workloads are supported via
20	ComputeClasses and NAP)
21	- `gcloud` CLI authenticated
22	- Sufficient GPU/TPU quota in the target region
23
24	## Workflow
25
26	### 1. Discovery: Find Models and Hardware
27
28	```bash
29	# List all supported models
30	gcloud container ai profiles models list --quiet
31
32	# Find valid accelerator/server combinations for a model
33	gcloud container ai profiles list --model=<MODEL_NAME> --quiet
34
35	# Example: what can run Gemma 2 9B?
36	gcloud container ai profiles list --model=gemma-2-9b-it --quiet
37	```
38
39	### 2. Generate Manifest
40
41	```bash
42	gcloud container ai profiles manifests create \
43	--model=<MODEL_NAME> \
44	--model-server=<SERVER> \
45	--accelerator-type=<ACCELERATOR> \
46	--target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
47	```
48
49	Parameters:
50
51	- `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`)
52	- `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`)
53	- `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`,
54	`nvidia-h100-80gb`)
55	- `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token
56	(optional, for latency optimization)
57
58	Example:
59
60	```bash
61	gcloud container ai profiles manifests create \
62	--model=gemma-2-9b-it \
63	--model-server=vllm \
64	--accelerator-type=nvidia-l4 \
65	--target-ntpot-milliseconds=50 --quiet > inference.yaml
66	```
67
68	### 3. Review and Deploy
69
70	```bash
71	# Review for placeholders (HF tokens, PVCs)
72	cat inference.yaml
73
74	# Deploy
75	kubectl apply -f inference.yaml
76
77	# Monitor
78	kubectl get pods -w
79	kubectl logs -f <POD_NAME>
80	```
81
82	> Some models require Hugging Face tokens. Create a Kubernetes Secret and
83	> reference it in the manifest.
84
85	## GPU ComputeClass for Inference
86
87	For Autopilot clusters, create a ComputeClass to target GPU nodes:
88
89	```yaml
90	apiVersion: cloud.google.com/v1
91	kind: ComputeClass
92	metadata:
93	name: l4-inference
94	spec:
95	priorities:
96	- machineFamily: g2
97	gpu:
98	type: nvidia-l4
99	count: 1
100	minCores: 4
101	minMemoryGb: 16
102	```
103
104	## Accelerator Selection Guide
105
106	\| Accelerator \| Best For \| Memory \| Relative Cost \|
107	\| ------------------- \| ------------------------ \| ----------- \| ------------- \|
108	\| NVIDIA T4 \| Budget inference, \| 16 GB \| Lowest \|
109	: : lightweight legacy : : :
110	: : models : : :
111	\| NVIDIA L4 (G2) \| Small-medium model \| 24 GB \| Low \|
112	: : inference, video, : : :
113	: : graphics : : :
114	\| NVIDIA RTX PRO 6000 \| Multimodal AI, \| 96 GB \| Medium \|
115	: (G4) : high-fidelity 3D, : : :
116	: : fine-tuning : : :
117	\| Cloud TPU v5e \| Cost-effective \| Varies \| Medium \|
118	: : transformer inference : : :
119	\| Cloud TPU v5p \| High-performance \| Varies \| High \|
120	: : training : : :
121	\| Cloud TPU v6e \| High-efficiency next-gen \| 32 GB/chip \| Medium-High \|
122	: (Trillium) : training & serving : : :
123	\| Cloud TPU v7x \| Ultra-scale inference & \| 192 GB/chip \| High \|
124	: (Ironwood) : agentic workflows : : :
125	\| NVIDIA A100 \| Large model inference, \| 40/80 GB \| High \|
126	: : enterprise ML : : :
127	\| NVIDIA H100 / H200 \| Frontier model training, \| 80/141 GB \| Highest \|
128	: : high throughput : : :
129	\| NVIDIA B200 (A4) \| Blackwell-scale \| 192 GB \| Highest \|
130	: : tr

People who liked this also viewed

6 picks

Type