perceiving-objects

Name: perceiving-objects
Author: graph-robots

26stars4forks

How to install perceiving-objects?

graph-robots/open-robot-skills/perceiving-objects

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-objects

Installs into the current project.

Summary

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented b

Files · 1

View on GitHub

SKILL.md

1	# perceiving-objects
2
3	Single-path perception: detect → disambiguate (pairwise crop tournament)
4	→ segment → fuse to 3D → extract OBB. Each DINO detection is cropped and
5	upscaled, and the target is found via binary "A or B?" comparisons of
6	crop pairs — far more reliable on small targets than a one-shot
7	Set-of-Marks letter pick (~30% → 97% on the LIBERO-PosVar object-ID
8	study).
9
10	Multi-camera handling uses a `safe` wrist-fallback gate (not blind
11	KD-tree fusion): identification defaults to the exterior view; the
12	wrist (eye-in-hand) view is consulted ONLY when the exterior pick fails
13	its own close-up verify AND the wrist pick passes its own. On the
14	4-suite / 200-frame regression study this was the only zero-regression
15	policy (+2.5% net, 0/189 frames regressed; blind fuse/verify→wrist/
16	wrist-only all regressed). See `perceive_dino_vlm.run` docstring.
17
18	On the verified-exterior path the wrist views still contribute **cloud
19	geometry** (never identity): wrist clouds of the same object — gated by
20	the multiview intersection check, with a geometry-seeded SAM fallback
21	(exterior cloud projected into the wrist frame) — are fused into the
22	output cloud so the OBB recovers the top face / far side a single front
23	view misses. A lone front view yields a sliver OBB biased toward the
24	camera by half the object depth, and that off-centre pinch is the
25	measured slip-during-transport failure mode on tall bottles/cartons.
26
27	## When to use
28
29	- Uncluttered scenes with visually distinct targets.
30	- Platforms where only DINO + VLM + SAM3 + geometry are deployed.
31	- The default single-target 3D perception skill.
32
33	## When NOT to use
34
35	- Cluttered scenes with many similar nearby distractors: strengthen the
36	pairwise tournament by passing `object_description` shape/appearance hints
37	(see the note above) rather than relying on the bare label alone.
38	- Clean-all-items / multi-item loops that need a clean "no match" loop
39	terminator. Prefer `perceiving-objects-oneshot`.
40
41	## Recommended subgraph state flow
42
43	3 states:
44
45	```text
46	observe → perceive → filter_obb
47	```
48
49	State details:
50
51	> About `object_name` below: it is a literal Python string — the natural
52	> noun phrase for the object you are perceiving, drawn from this subgraph's
53	> description (e.g. `"alphabet soup can"`, `"basket"`, `"red bowl"`). It is
54	> a constant per subgraph instance, NOT a binding. DO NOT write
55	> `Ref("in.object_name")` or any other `Ref(...)`; the coordinator does
56	> not declare `object_name` as a subgraph input. Write the string directly,
57	> e.g. `"object_name": "basket"`.
58	>
59	> About `object_description` (wire it whenever the task gives hints):
60	> also a literal Python string. When the task/workflow description carries
61	> shape or appearance hints for the target (e.g. an "Object context" block
62	> with `shape_hint` / `expected_label`, or adjectives in the instruction),
63	> pass them throu

People who liked this also viewed

6 picks

Type

perceiving-objects

26stars4forks

View on GitHub

How to install perceiving-objects?

graph-robots/open-robot-skills/perceiving-objects

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-objects

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# perceiving-objects
2
3	Single-path perception: detect → disambiguate (pairwise crop tournament)
4	→ segment → fuse to 3D → extract OBB. Each DINO detection is cropped and
5	upscaled, and the target is found via binary "A or B?" comparisons of
6	crop pairs — far more reliable on small targets than a one-shot
7	Set-of-Marks letter pick (~30% → 97% on the LIBERO-PosVar object-ID
8	study).
9
10	Multi-camera handling uses a `safe` wrist-fallback gate (not blind
11	KD-tree fusion): identification defaults to the exterior view; the
12	wrist (eye-in-hand) view is consulted ONLY when the exterior pick fails
13	its own close-up verify AND the wrist pick passes its own. On the
14	4-suite / 200-frame regression study this was the only zero-regression
15	policy (+2.5% net, 0/189 frames regressed; blind fuse/verify→wrist/
16	wrist-only all regressed). See `perceive_dino_vlm.run` docstring.
17
18	On the verified-exterior path the wrist views still contribute **cloud
19	geometry** (never identity): wrist clouds of the same object — gated by
20	the multiview intersection check, with a geometry-seeded SAM fallback
21	(exterior cloud projected into the wrist frame) — are fused into the
22	output cloud so the OBB recovers the top face / far side a single front
23	view misses. A lone front view yields a sliver OBB biased toward the
24	camera by half the object depth, and that off-centre pinch is the
25	measured slip-during-transport failure mode on tall bottles/cartons.
26
27	## When to use
28
29	- Uncluttered scenes with visually distinct targets.
30	- Platforms where only DINO + VLM + SAM3 + geometry are deployed.
31	- The default single-target 3D perception skill.
32
33	## When NOT to use
34
35	- Cluttered scenes with many similar nearby distractors: strengthen the
36	pairwise tournament by passing `object_description` shape/appearance hints
37	(see the note above) rather than relying on the bare label alone.
38	- Clean-all-items / multi-item loops that need a clean "no match" loop
39	terminator. Prefer `perceiving-objects-oneshot`.
40
41	## Recommended subgraph state flow
42
43	3 states:
44
45	```text
46	observe → perceive → filter_obb
47	```
48
49	State details:
50
51	> About `object_name` below: it is a literal Python string — the natural
52	> noun phrase for the object you are perceiving, drawn from this subgraph's
53	> description (e.g. `"alphabet soup can"`, `"basket"`, `"red bowl"`). It is
54	> a constant per subgraph instance, NOT a binding. DO NOT write
55	> `Ref("in.object_name")` or any other `Ref(...)`; the coordinator does
56	> not declare `object_name` as a subgraph input. Write the string directly,
57	> e.g. `"object_name": "basket"`.
58	>
59	> About `object_description` (wire it whenever the task gives hints):
60	> also a literal Python string. When the task/workflow description carries
61	> shape or appearance hints for the target (e.g. an "Object context" block
62	> with `shape_hint` / `expected_label`, or adjectives in the instruction),
63	> pass them throu

People who liked this also viewed

6 picks

Type