perceiving-object-parts

Name: perceiving-object-parts
Author: graph-robots

26stars4forks

How to install perceiving-object-parts?

graph-robots/open-robot-skills/perceiving-object-parts

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-object-parts

Installs into the current project.

Summary

Hierarchical perception for subpart targeting. Detects a parent object first (DINO+VLM), crops the camera image to the parent's bounding box, then detects and segments the named subpart inside the crop (DINO + SAM3), and uncrops + fuses depth to a world-frame OBB/mask/cloud — plu

Files · 1

View on GitHub

SKILL.md

1	# perceiving-object-parts
2
3	Two-step zoom-in perception. The full image gives a small subpart
4	(e.g. a frypan handle is ~3% of pixels) bad signal-to-noise for SAM3
5	text segmentation; cropping to the parent first brings the subpart up
6	to ~30% of pixels in the cropped image — within SAM3's reliable range.
7
8	> About `parent_prompt` and `subpart_prompt`: they are **literal
9	> Python strings**, NOT subgraph inputs. They are author-time constants
10	> per subgraph instance. DO NOT declare them in the subgraph's
11	> top-level `inputs` block, and DO NOT write `Ref("in.parent_prompt")`
12	> or any other `Ref(...)` for them. Write the strings directly on the
13	> inner script node, e.g.
14	> `"parent_prompt": "frying pan", "subpart_prompt": "long horizontal handle of the frying pan"`.
15	> Only `cameras` is a flowed subgraph input (wired from the workflow's
16	> observation source, identical to `perceiving-objects`'s `cameras`
17	> input).
18
19	## When to use
20
21	- The grasp/place affordance is a part of a larger object (pan handle,
22	drawer pull, moka-pot grip, mug rim, stove burner).
23	- Plain `perceiving-objects` with `object_name="handle"` fails because
24	there are multiple handles in the scene (drawer pull, microwave
25	door, cabinet, ...) and DINO can't disambiguate.
26
27	## When NOT to use
28
29	- The whole object IS the target (`perceiving-objects` is faster and
30	produces a cleaner OBB).
31	- The subpart spans the majority of the image already (skip the crop).
32
33	## Pipeline
34
35	```
36	observation # rgb + depth + intrinsics + camera pose
37	│
38	▼ grounding-dino.detect(rgb, parent_prompt)
39	parent_box (BoundingBox2D) # broadest of the boxes, or VLM-picked
40	│
41	▼ crop_rgb_to_box(parent_box, padding=30)
42	cropped_image # H_new × W_new × 3 uint8
43	│
44	▼ grounding-dino.detect

People who liked this also viewed

6 picks

Type

perceiving-object-parts

26stars4forks

View on GitHub

How to install perceiving-object-parts?

graph-robots/open-robot-skills/perceiving-object-parts

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-object-parts

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# perceiving-object-parts
2
3	Two-step zoom-in perception. The full image gives a small subpart
4	(e.g. a frypan handle is ~3% of pixels) bad signal-to-noise for SAM3
5	text segmentation; cropping to the parent first brings the subpart up
6	to ~30% of pixels in the cropped image — within SAM3's reliable range.
7
8	> About `parent_prompt` and `subpart_prompt`: they are **literal
9	> Python strings**, NOT subgraph inputs. They are author-time constants
10	> per subgraph instance. DO NOT declare them in the subgraph's
11	> top-level `inputs` block, and DO NOT write `Ref("in.parent_prompt")`
12	> or any other `Ref(...)` for them. Write the strings directly on the
13	> inner script node, e.g.
14	> `"parent_prompt": "frying pan", "subpart_prompt": "long horizontal handle of the frying pan"`.
15	> Only `cameras` is a flowed subgraph input (wired from the workflow's
16	> observation source, identical to `perceiving-objects`'s `cameras`
17	> input).
18
19	## When to use
20
21	- The grasp/place affordance is a part of a larger object (pan handle,
22	drawer pull, moka-pot grip, mug rim, stove burner).
23	- Plain `perceiving-objects` with `object_name="handle"` fails because
24	there are multiple handles in the scene (drawer pull, microwave
25	door, cabinet, ...) and DINO can't disambiguate.
26
27	## When NOT to use
28
29	- The whole object IS the target (`perceiving-objects` is faster and
30	produces a cleaner OBB).
31	- The subpart spans the majority of the image already (skip the crop).
32
33	## Pipeline
34
35	```
36	observation # rgb + depth + intrinsics + camera pose
37	│
38	▼ grounding-dino.detect(rgb, parent_prompt)
39	parent_box (BoundingBox2D) # broadest of the boxes, or VLM-picked
40	│
41	▼ crop_rgb_to_box(parent_box, padding=30)
42	cropped_image # H_new × W_new × 3 uint8
43	│
44	▼ grounding-dino.detect

People who liked this also viewed

6 picks

Type