perceiving-objects-oneshot

Name: perceiving-objects-oneshot
Author: graph-robots

26stars4forks

How to install perceiving-objects-oneshot?

graph-robots/open-robot-skills/perceiving-objects-oneshot

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-objects-oneshot

Installs into the current project.

Summary

Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Return

Files · 1

View on GitHub

SKILL.md

1	# perceiving-objects-oneshot
2
3	Single VLM call over a set-of-marks overlay. Pipeline:
4
5	```text
6	observe → perceive → filter_obb
7	```
8
9	`perceive` runs:
10
11	1. ``grounding-dino.detect`` with a broad ``object.`` text prompt
12	2. One ``vlm.query`` showing the image with letter-labeled boxes:
13	"Which letter is the <target>? Reply with one letter or 'none'."
14	3. On ``none``: emit ``found: False`` so the subgraph exits
15	``not_found``. On a letter: ``sam3.segment_box`` on the chosen box,
16	``geometry.mask_to_world_points`` for the cloud.
17
18	## When to use
19
20	- Clean-all-items / multi-item loops where the cycle needs a clean
21	"no match" signal to terminate via ``target.not_found → done``.
22	- Tasks where the target description is generic ("any item on the
23	floor", "the next remaining grocery item") rather than a specific
24	scene-spec id.
25	- Uncluttered scenes with distinct, reasonably sized targets where the
26	set-of-marks letter pick is reliable.
27
28	## When NOT to use
29
30	- Small / cluttered targets (< 40 px wide) — prefer ``perceiving-objects``
31	whose pairwise crop tournament is far more reliable in that regime.
32
33	## Recommended subgraph state flow
34
35	3 states: ``observe → perceive → filter_obb`` (mirrors ``perceiving-objects``).
36
37	State details:
38
39	> About `object_name` below: it is a literal Python string — the
40	> natural noun phrase for the object you are perceiving, drawn from this
41	> subgraph's description (e.g. `"alphabet soup"`, `"basket"`,
42	> `"any grocery item on the floor"`). It is a constant per subgraph
43	> instance, NOT a binding. DO NOT write `Ref("in.object_name")` or
44	> any other `Ref(...)`; the coordinator does not declare `object_name`
45	> as a subgraph input. Write the string directly,
46	> e.g. `"object_name": "any grocery item on the floor"`.
47	> The same rule applies to `object_description` if you set it.
48
49	1. `observe` — `type: tool`, `tool: "robot.get_observation"`,
50	`inputs: {}`. Connector tool; flat name only.
51	2. `perceive` — `type: script`, file `scripts/<sg>/perceive_simple.py`
52	from this bundle. Inputs:
53	`cameras=Ref("observe.cameras")`,
54	`object_name="<noun phrase from the subgraph description>"`,
55	plus any optional literals (`object_description`, `dino_prompt`).
56	Returns `{found, cloud, mask, score}`. When the VLM picks "none" or
57	DINO emits no detections, ``found`` is `False` and the downstream
58	`filter_obb` step then raises (empty cloud) — caught by the
59	subgraph's `on_error: "not_found"` exit.
60	3. `filter_obb` — `type: tool`,
61	`tool: "geometry.filter_and_compute_obb"`,
62	`inputs={"points": Ref("perceive.cloud")}`. Returns
63	`{"obb": <OrientedBoundingBox>}`.
64
65	### Wiring the exit (HARD)
66
67	Linear `perceive → filter_obb → found → END`. The `filter_obb` tool
68	raises on empty clouds (the not-found path), and the subgraph's
69	`on_error: "not_found"` catches that. Do NOT add

People who liked this also viewed

6 picks

Type

perceiving-objects-oneshot

26stars4forks

View on GitHub

How to install perceiving-objects-oneshot?

graph-robots/open-robot-skills/perceiving-objects-oneshot

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-objects-oneshot

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# perceiving-objects-oneshot
2
3	Single VLM call over a set-of-marks overlay. Pipeline:
4
5	```text
6	observe → perceive → filter_obb
7	```
8
9	`perceive` runs:
10
11	1. ``grounding-dino.detect`` with a broad ``object.`` text prompt
12	2. One ``vlm.query`` showing the image with letter-labeled boxes:
13	"Which letter is the <target>? Reply with one letter or 'none'."
14	3. On ``none``: emit ``found: False`` so the subgraph exits
15	``not_found``. On a letter: ``sam3.segment_box`` on the chosen box,
16	``geometry.mask_to_world_points`` for the cloud.
17
18	## When to use
19
20	- Clean-all-items / multi-item loops where the cycle needs a clean
21	"no match" signal to terminate via ``target.not_found → done``.
22	- Tasks where the target description is generic ("any item on the
23	floor", "the next remaining grocery item") rather than a specific
24	scene-spec id.
25	- Uncluttered scenes with distinct, reasonably sized targets where the
26	set-of-marks letter pick is reliable.
27
28	## When NOT to use
29
30	- Small / cluttered targets (< 40 px wide) — prefer ``perceiving-objects``
31	whose pairwise crop tournament is far more reliable in that regime.
32
33	## Recommended subgraph state flow
34
35	3 states: ``observe → perceive → filter_obb`` (mirrors ``perceiving-objects``).
36
37	State details:
38
39	> About `object_name` below: it is a literal Python string — the
40	> natural noun phrase for the object you are perceiving, drawn from this
41	> subgraph's description (e.g. `"alphabet soup"`, `"basket"`,
42	> `"any grocery item on the floor"`). It is a constant per subgraph
43	> instance, NOT a binding. DO NOT write `Ref("in.object_name")` or
44	> any other `Ref(...)`; the coordinator does not declare `object_name`
45	> as a subgraph input. Write the string directly,
46	> e.g. `"object_name": "any grocery item on the floor"`.
47	> The same rule applies to `object_description` if you set it.
48
49	1. `observe` — `type: tool`, `tool: "robot.get_observation"`,
50	`inputs: {}`. Connector tool; flat name only.
51	2. `perceive` — `type: script`, file `scripts/<sg>/perceive_simple.py`
52	from this bundle. Inputs:
53	`cameras=Ref("observe.cameras")`,
54	`object_name="<noun phrase from the subgraph description>"`,
55	plus any optional literals (`object_description`, `dino_prompt`).
56	Returns `{found, cloud, mask, score}`. When the VLM picks "none" or
57	DINO emits no detections, ``found`` is `False` and the downstream
58	`filter_obb` step then raises (empty cloud) — caught by the
59	subgraph's `on_error: "not_found"` exit.
60	3. `filter_obb` — `type: tool`,
61	`tool: "geometry.filter_and_compute_obb"`,
62	`inputs={"points": Ref("perceive.cloud")}`. Returns
63	`{"obb": <OrientedBoundingBox>}`.
64
65	### Wiring the exit (HARD)
66
67	Linear `perceive → filter_obb → found → END`. The `filter_obb` tool
68	raises on empty clouds (the not-found path), and the subgraph's
69	`on_error: "not_found"` catches that. Do NOT add

People who liked this also viewed

6 picks

Type