dstack-prototyping

Name: dstack-prototyping
Author: dstackai

2.2kstars239forks

How to install dstack-prototyping?

dstackai/dstack/dstack-prototyping

$npx -y skills add dstackai/dstack --skill dstack-prototyping

Installs into the current project.

Summary

Use with the dstack skill for model-serving work when the image, serving command, resources, backend/fleet choice, or service behavior is not proven. Guides task-first prototyping on real hardware, choosing fleets/backends that can reuse idle instances and caches, checking vLLM/S

Files · 1

View on GitHub

SKILL.md

1	# dstack Prototyping
2
3	Use `/dstack` for CLI commands, YAML fields, apply/attach behavior, service URLs,
4	and other dstack syntax. This skill explains how to use dstack runs while the
5	model-serving configuration is still unknown.
6
7	## Goal
8
9	Find a working dstack service configuration for the requested model.
10
11	Before submitting a service, use a task on real hardware to test the serving
12	image, install/runtime assumptions, model download, cache path, command, port,
13	launch flags, resources, env vars, backend/fleet choice, and local model
14	request. Then submit the same configuration as a service and verify the model
15	through the dstack service URL.
16
17	## Choose Where To Run
18
19	Choose only VM-based backends, SSH fleets, or Kubernetes fleets because they support idle instances and/or instance volumes. That lets later runs reuse the provisioned/idle instance or instance volumes used by runs for caching model weights (and possibly other writes). You must follow this rule even if there are fleets/backends/offers that are cheaper. The only exception from this rule is when the required GPU class (regardless of the price) is not available through VM-based backend, SSH fleet, or Kubernetes fleet.
20
21	Read `https://dstack.ai/docs/concepts/backends.md` to know exactly which
22	backends are VM-based.
23
24	## Check Serving Sources
25
26	Check serving-framework sources early enough to choose the image, command,
27	launch flags, resources, cache paths, request format, and expected model
28	behavior.
29
30	For vLLM and SGLang, use these as credible sources:
31
32	- vLLM recipes and model index: `https://recipes.vllm.ai/` and
33	`https://recipes.vllm.ai/models.json`
34	- vLLM recipe docs: `https://docs.vllm.ai/projects/recipes/en/stable/`
35	- SGLang docs and cookbook: `https://docs.sglang.ai/` and
36	`https://lmsysorg.mintlify.app/cookbook/intro`
37
38	Use deeper serving-engine writeups, such as
39	`https://www.lmsys.org/blog/2026-07-02-agent-assisted-sglang-development`, when
40	these references do not explain the model, hardware, or serving failure.
41
42	## Use A Task Before Service
43
44	Before submitting a service, start a long-lived task:
45
46	```yaml
47	commands:
48	- sleep infinity
49	```
50
51	or an equivalent idle command.
52
53	Submit the task detached, attach or SSH into it when available, and run commands
54	inside the live environment. Test the image, installs, model download and cache
55	path, serving command, port, launch flags, local model request, and expected
56	model behavior.
57
58	When starting a long-running command in the background from a non-interactive
59	SSH command, use `nohup`, redirect stdin from `/dev/null`, and redirect
60	stdout/stderr to a log file so the SSH command returns while the process keeps
61	running. For example (the command can be any long-running command):
62
63	```shell
64	nohup vllm serve ... </dev/null > /tmp/vllm.log 2>&1 &
65	```
66
67	If the image, hardware choice, or major install path changes, submit another
68	task so the changed setup is tested before service verification.
69
70	Do not move to a service after checking only GPU visibility, imports, logs, or a
71	health endpoint. Start the server inside the task and send a request that uses
72	the requested model. For a chat or reasoning model, check the response behavior
73	the endpoint is expected to support, such as reasoning output when that model is
74	supposed to expose it.
75
76	Follow `/dstack` structured status guidance when polling task or service status.
77	After requesting a task or service stop before another submission, wait until
78	that run reaches a terminal status. This allows dstack to reuse its instance or
79	instance volumes when available.
80
81	## Verify As A Service
82
83	Submit the service after the task has verified the configuration: image,
84	command, port, resources, env vars, cache mounts if used, backend/fleet choice,
85	and model request.
86
87	Use the service as a duplicate check of the same configuration under dstack
88	service runtime. The model request that worked locally in the task must also work
89	through the dstack service URL.
90
91	If service verification fails because the image, install, model download,
92	command, resources, cache, or model behavior needs to change, go back to a task.
93	If the tested serving setup is still right and only the dstack service
94	configuration is wrong, fix the configuration and submit the service again.

People who liked this also viewed

6 picks

Type

dstack-prototyping

2.2kstars239forks

View on GitHub

How to install dstack-prototyping?

dstackai/dstack/dstack-prototyping

$npx -y skills add dstackai/dstack --skill dstack-prototyping

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# dstack Prototyping
2
3	Use `/dstack` for CLI commands, YAML fields, apply/attach behavior, service URLs,
4	and other dstack syntax. This skill explains how to use dstack runs while the
5	model-serving configuration is still unknown.
6
7	## Goal
8
9	Find a working dstack service configuration for the requested model.
10
11	Before submitting a service, use a task on real hardware to test the serving
12	image, install/runtime assumptions, model download, cache path, command, port,
13	launch flags, resources, env vars, backend/fleet choice, and local model
14	request. Then submit the same configuration as a service and verify the model
15	through the dstack service URL.
16
17	## Choose Where To Run
18
19	Choose only VM-based backends, SSH fleets, or Kubernetes fleets because they support idle instances and/or instance volumes. That lets later runs reuse the provisioned/idle instance or instance volumes used by runs for caching model weights (and possibly other writes). You must follow this rule even if there are fleets/backends/offers that are cheaper. The only exception from this rule is when the required GPU class (regardless of the price) is not available through VM-based backend, SSH fleet, or Kubernetes fleet.
20
21	Read `https://dstack.ai/docs/concepts/backends.md` to know exactly which
22	backends are VM-based.
23
24	## Check Serving Sources
25
26	Check serving-framework sources early enough to choose the image, command,
27	launch flags, resources, cache paths, request format, and expected model
28	behavior.
29
30	For vLLM and SGLang, use these as credible sources:
31
32	- vLLM recipes and model index: `https://recipes.vllm.ai/` and
33	`https://recipes.vllm.ai/models.json`
34	- vLLM recipe docs: `https://docs.vllm.ai/projects/recipes/en/stable/`
35	- SGLang docs and cookbook: `https://docs.sglang.ai/` and
36	`https://lmsysorg.mintlify.app/cookbook/intro`
37
38	Use deeper serving-engine writeups, such as
39	`https://www.lmsys.org/blog/2026-07-02-agent-assisted-sglang-development`, when
40	these references do not explain the model, hardware, or serving failure.
41
42	## Use A Task Before Service
43
44	Before submitting a service, start a long-lived task:
45
46	```yaml
47	commands:
48	- sleep infinity
49	```
50
51	or an equivalent idle command.
52
53	Submit the task detached, attach or SSH into it when available, and run commands
54	inside the live environment. Test the image, installs, model download and cache
55	path, serving command, port, launch flags, local model request, and expected
56	model behavior.
57
58	When starting a long-running command in the background from a non-interactive
59	SSH command, use `nohup`, redirect stdin from `/dev/null`, and redirect
60	stdout/stderr to a log file so the SSH command returns while the process keeps
61	running. For example (the command can be any long-running command):
62
63	```shell
64	nohup vllm serve ... </dev/null > /tmp/vllm.log 2>&1 &
65	```
66
67	If the image, hardware choice, or major install path changes, submit another
68	task so the changed setup is tested before service verification.
69
70	Do not move to a service after checking only GPU visibility, imports, logs, or a
71	health endpoint. Start the server inside the task and send a request that uses
72	the requested model. For a chat or reasoning model, check the response behavior
73	the endpoint is expected to support, such as reasoning output when that model is
74	supposed to expose it.
75
76	Follow `/dstack` structured status guidance when polling task or service status.
77	After requesting a task or service stop before another submission, wait until
78	that run reaches a terminal status. This allows dstack to reuse its instance or
79	instance volumes when available.
80
81	## Verify As A Service
82
83	Submit the service after the task has verified the configuration: image,
84	command, port, resources, env vars, cache mounts if used, backend/fleet choice,
85	and model request.
86
87	Use the service as a duplicate check of the same configuration under dstack
88	service runtime. The model request that worked locally in the task must also work
89	through the dstack service URL.
90
91	If service verification fails because the image, install, model download,
92	command, resources, cache, or model behavior needs to change, go back to a task.
93	If the tested serving setup is still right and only the dstack service
94	configuration is wrong, fix the configuration and submit the service again.

People who liked this also viewed

6 picks

Type