sam3

Name: sam3
Author: graph-robots

26stars4forks

How to install sam3?

graph-robots/open-robot-skills/sam3

$npx -y skills add graph-robots/open-robot-skills --skill sam3

Installs into the current project.

Summary

Segment Anything 3 — text-, point-, and box-prompted instance

Files · 1

View on GitHub

SKILL.md

1	# sam3
2
3	The SAM3 image servicer + video-tracker servicer as in-process tools. Images
4	are RGB uint8 `[H, W, 3]` numpy arrays; masks come back as gap `Mask`
5	(uint8 `[H, W]`, 0 background / 255 foreground), score-sorted best-first.
6
7	## When to use
8
9	- `segment_text` for open-vocabulary "find the X" masks (one mask per
10	instance; check `scores[0]` — callers typically reject below ~0.3).
11	- `segment_box` after a detector (e.g. `grounding-dino.detect`) for a
12	pixel-accurate mask inside the detection box; add the point prompt
13	(`use_point=True`) when a pointing model supplies one.
14	- `tracker_init` / `tracker_update` / `tracker_close` to follow a single
15	target across an observation stream (e.g. for visual servoing).
16
17	## Install
18
19	```bash
20	uv sync --extra sam3 # torch + torchvision + the upstream sam3 package
21	# (pip: pip install -e ".[sam3]")
22	```
23
24	Model weights download on first model build. Device is taken from
25	`GAP_SAM3_DEVICE` (default `cuda`); the image model also runs on `cpu`
26	(slow), the video tracker is CUDA-only in practice.
27
28	## Gotchas (carried over from the servicers)
29
30	- Lazy singletons: the image model and the video predictor each load on
31	first call and stay resident; importing the bundle never imports torch.
32	- `segment_text` caps results at `max_results=5` by default — cluttered
33	scenes emit 100+ instances (~1 MB/mask at 720p) and downstream consumes
34	only the top mask. Pass `max_results<=0` for everything.
35	- The video tracker JIT-compiles Triton NMS kernels via the `CC` env
36	var; a stale `CC` (e.g. a Ray env pointing at a non-existent gcc-13)
37	surfaces as `FileNotFoundError` inside `tracker_init`. The bundle forces
38	`CC` to a real compiler before tracker use (`_ensure_cc_compiler`).
39	- Tracker prompt precedence is box > point > text; a point prompt is
40	converted to a small (10% of image) box because the predictor's box path
41	is more reliable for init than a single point.
42	- The tracker is built with `apply_temporal_disambiguation=False` — the
43	default hotstart heuristics silently delete the masklet around frame 3 in
44	streaming mode (no fresh text re-detection per frame).
45	- Drift handling in `tracker_update`: a mask-area jump >1.5x the running
46	median or confidence <0.30 keeps the LAST GOOD mask and reports
47	`confidence=0.0` with `object_present=True` (skip this frame); after 5
48	consecutive drift hits `object_present=False` — re-init the tracker.
49	- Sessions idle longer than 120 s are evicted lazily on the next tracker
50	call; an evicted/unknown `tracker_id` raises `ToolError`.
51	- `tracker_init` returns `object_present=False` with an empty `tracker_id`
52	(no exception) when the initial detection finds nothing.

People who liked this also viewed

6 picks

Type

find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 20262.6M27k
grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026633k182k
vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026573k29k
improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026522k182k
setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026454k182k
triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026424k182k

sam3

26stars4forks

View on GitHub

How to install sam3?

graph-robots/open-robot-skills/sam3

$npx -y skills add graph-robots/open-robot-skills --skill sam3

Installs into the current project.

Summary

Segment Anything 3 — text-, point-, and box-prompted instance

Files · 1

View on GitHub

SKILL.md

1	# sam3
2
3	The SAM3 image servicer + video-tracker servicer as in-process tools. Images
4	are RGB uint8 `[H, W, 3]` numpy arrays; masks come back as gap `Mask`
5	(uint8 `[H, W]`, 0 background / 255 foreground), score-sorted best-first.
6
7	## When to use
8
9	- `segment_text` for open-vocabulary "find the X" masks (one mask per
10	instance; check `scores[0]` — callers typically reject below ~0.3).
11	- `segment_box` after a detector (e.g. `grounding-dino.detect`) for a
12	pixel-accurate mask inside the detection box; add the point prompt
13	(`use_point=True`) when a pointing model supplies one.
14	- `tracker_init` / `tracker_update` / `tracker_close` to follow a single
15	target across an observation stream (e.g. for visual servoing).
16
17	## Install
18
19	```bash
20	uv sync --extra sam3 # torch + torchvision + the upstream sam3 package
21	# (pip: pip install -e ".[sam3]")
22	```
23
24	Model weights download on first model build. Device is taken from
25	`GAP_SAM3_DEVICE` (default `cuda`); the image model also runs on `cpu`
26	(slow), the video tracker is CUDA-only in practice.
27
28	## Gotchas (carried over from the servicers)
29
30	- Lazy singletons: the image model and the video predictor each load on
31	first call and stay resident; importing the bundle never imports torch.
32	- `segment_text` caps results at `max_results=5` by default — cluttered
33	scenes emit 100+ instances (~1 MB/mask at 720p) and downstream consumes
34	only the top mask. Pass `max_results<=0` for everything.
35	- The video tracker JIT-compiles Triton NMS kernels via the `CC` env
36	var; a stale `CC` (e.g. a Ray env pointing at a non-existent gcc-13)
37	surfaces as `FileNotFoundError` inside `tracker_init`. The bundle forces
38	`CC` to a real compiler before tracker use (`_ensure_cc_compiler`).
39	- Tracker prompt precedence is box > point > text; a point prompt is
40	converted to a small (10% of image) box because the predictor's box path
41	is more reliable for init than a single point.
42	- The tracker is built with `apply_temporal_disambiguation=False` — the
43	default hotstart heuristics silently delete the masklet around frame 3 in
44	streaming mode (no fresh text re-detection per frame).
45	- Drift handling in `tracker_update`: a mask-area jump >1.5x the running
46	median or confidence <0.30 keeps the LAST GOOD mask and reports
47	`confidence=0.0` with `object_present=True` (skip this frame); after 5
48	consecutive drift hits `object_present=False` — re-init the tracker.
49	- Sessions idle longer than 120 s are evicted lazily on the next tracker
50	call; an evicted/unknown `tracker_id` raises `ToolError`.
51	- `tracker_init` returns `object_present=False` with an empty `tracker_id`
52	(no exception) when the initial detection finds nothing.

People who liked this also viewed

6 picks

Type