perceiving-next-item

Name: perceiving-next-item
Author: graph-robots

26stars4forks

How to install perceiving-next-item?

graph-robots/open-robot-skills/perceiving-next-item

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-next-item

Installs into the current project.

Summary

--- name: perceiving-next-item description: > Loop-head perception for pack-all / clean-all-items tasks. From ONE observation it localizes BOTH the destination container (basket, bin, box) AND the next remaining target item, using the pairwise VLM crop tournament with a c

Files · 1

View on GitHub

SKILL.md

1	---
2	name: perceiving-next-item
3	description: >
4	Loop-head perception for pack-all / clean-all-items tasks. From ONE
5	observation it localizes BOTH the destination container (basket, bin, box)
6	AND the next remaining target item, using the pairwise VLM crop tournament
7	with a container-excluding description so the target is never confused with
8	the basket, then makes a clean found / none decision: an item was found
9	(grasp it) or only the container remains (all items packed — exit the loop
10	to done). Use when a workflow must pick up EVERY object and place each into a
11	container in a loop (pack-all / clean-all-items) and each pass must reliably
12	answer "is there still a graspable item, or are we done?" while also exposing
13	a fresh container OBB for the downstream place. This subgraph is
14	self-contained and takes NO inputs (inputs: {}); the container and item
15	phrases are literal strings written inside the perception nodes, never
16	subgraph parameters — do not declare item_name / container_name /
17	item_description as subgraph inputs.
18	license: MIT
19	compatibility: requires gap>=0.1
20	metadata:
21	category: perception
22	tags: [perception, dino, vlm, sam3, loop, pack-all, clean-all, container]
23	gap:
24	allowed_tools:
25	- robot.get_observation
26	- grounding-dino.detect
27	- vlm.query
28	- vlm.query_yes_no
29	- sam3.segment_box
30	- sam3.segment_text
31	- geometry.mask_to_world_points
32	- geometry.exclude_robot_points
33	- geometry.filter_and_compute_obb
34	exit_conditions:
35	found: A graspable target item distinct from the container was localized; target_obb/mask and container_obb/mask/cloud bound in the subgraph outputs. Route to a grasp skill.
36	none: No target item remains — only the container is left, so every item is packed. This is the clean loop terminator; route to done (NOT abort — it is a success).
37	perception_failed: Perception raised an actual error (empty detection is NOT this — that is the normal `none` path). Route to abort. Named so it is never confused with "nothing left"; do NOT route it to done.
38	produces_outputs:
39	target_obb: OrientedBoundingBox
40	target_mask: Mask
41	container_obb: OrientedBoundingBox
42	container_mask: Mask
43	container_cloud: PointCloud
44	hard_rules:
45	- perception_pipeline_invariants.md#emit-both-obb-and-mask
46	- geometry_calling_conventions.md#obb-field-binding
47	- >
48	The item perception MUST pass an `object_description` that names the
49	grocery item categories AND explicitly excludes the container (e.g.
50	"a packaged grocery product such as a can, box, carton, jar, or bottle.
51	Never the wicker basket or storage container."). This exclusion is what
52	makes the tournament return "no item" once only the basket remains — it
53	is the whole basis of the clean `none` loop-exit. Do NOT rely on
54	geometry to reject the basket.
55	- >
56	This subgraph declares NO inputs (`inputs: {}`) and is fully
57	self-contained. Every value that tells the perceive nodes WHAT to look
58	for is a LITERAL string hardcoded in the node — use exactly the node input
59	names `object_name` and `object_description` from
60	examples/canonical_subgraph.json (do not rename them to `item_name`,
61	`container_name`, `item_description`, etc.). DO NOT lift ANY of them to a
62	subgraph input and DO NOT write `Ref("in.<anything>")` for them: there is
63	no upstream producer, so it fails validation (W8: no upstream producer).
64	The basket and item names are known at authoring time — write them in.
65	- >
66	The `decide` router (scripts/<sg>/decide_next_item.py) sits BETWEEN
67	`perceive_item` and `filter_obb_item`, and maps `found -> filter_obb_item`,
68	`none -> none`. Declare `success_values: ["found", "none"]` so BOTH are
69	clean exits; only genuine perception errors fall through to
70	`on_error: perception_failed`.
71	- >
72	TOP-LEVEL ROUTING (do not get this wrong): map this subgraph's
73	`found -> grasp`, `none -> done` (all items packed — a SUCCESS), and
74	`perception_failed -> abort`. `none` is the loop terminator, NOT
75	`perception_failed`; never route `perception_failed` to done.
76	- >
77	Perceive the item from the raw `observe.cameras`, but perceive the
78	container through `exterior_view` (drop the wrist cam) — the angled
79	eye-in-hand view bloats the container OBB by fusing neighbouring points.
80	- >
81	WIRE A SINGLE CHAIN, never parallel branches: observe → exterior_view →
82	perceive_container → filter_obb_container → perceive_item → decide.
83	Do NOT add a second edge from `observe` straight to `perceive_item` to
84	"parallelize" the two perceptions: the runtime scheduler has no join
85	barrier, so `decide` (which consumes `filter_obb_container.obb`) can be
86	scheduled by `perceive_item`'s completion before `filter_obb_container`
87	has run — the $ref fails at runtime and the subgraph aborts on every
88	trial (validator rule S12 rej

People who liked this also viewed

6 picks

Type

perceiving-next-item

26stars4forks

View on GitHub

How to install perceiving-next-item?

graph-robots/open-robot-skills/perceiving-next-item

$npx -y skills add graph-robots/open-robot-skills --skill perceiving-next-item

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	---
2	name: perceiving-next-item
3	description: >
4	Loop-head perception for pack-all / clean-all-items tasks. From ONE
5	observation it localizes BOTH the destination container (basket, bin, box)
6	AND the next remaining target item, using the pairwise VLM crop tournament
7	with a container-excluding description so the target is never confused with
8	the basket, then makes a clean found / none decision: an item was found
9	(grasp it) or only the container remains (all items packed — exit the loop
10	to done). Use when a workflow must pick up EVERY object and place each into a
11	container in a loop (pack-all / clean-all-items) and each pass must reliably
12	answer "is there still a graspable item, or are we done?" while also exposing
13	a fresh container OBB for the downstream place. This subgraph is
14	self-contained and takes NO inputs (inputs: {}); the container and item
15	phrases are literal strings written inside the perception nodes, never
16	subgraph parameters — do not declare item_name / container_name /
17	item_description as subgraph inputs.
18	license: MIT
19	compatibility: requires gap>=0.1
20	metadata:
21	category: perception
22	tags: [perception, dino, vlm, sam3, loop, pack-all, clean-all, container]
23	gap:
24	allowed_tools:
25	- robot.get_observation
26	- grounding-dino.detect
27	- vlm.query
28	- vlm.query_yes_no
29	- sam3.segment_box
30	- sam3.segment_text
31	- geometry.mask_to_world_points
32	- geometry.exclude_robot_points
33	- geometry.filter_and_compute_obb
34	exit_conditions:
35	found: A graspable target item distinct from the container was localized; target_obb/mask and container_obb/mask/cloud bound in the subgraph outputs. Route to a grasp skill.
36	none: No target item remains — only the container is left, so every item is packed. This is the clean loop terminator; route to done (NOT abort — it is a success).
37	perception_failed: Perception raised an actual error (empty detection is NOT this — that is the normal `none` path). Route to abort. Named so it is never confused with "nothing left"; do NOT route it to done.
38	produces_outputs:
39	target_obb: OrientedBoundingBox
40	target_mask: Mask
41	container_obb: OrientedBoundingBox
42	container_mask: Mask
43	container_cloud: PointCloud
44	hard_rules:
45	- perception_pipeline_invariants.md#emit-both-obb-and-mask
46	- geometry_calling_conventions.md#obb-field-binding
47	- >
48	The item perception MUST pass an `object_description` that names the
49	grocery item categories AND explicitly excludes the container (e.g.
50	"a packaged grocery product such as a can, box, carton, jar, or bottle.
51	Never the wicker basket or storage container."). This exclusion is what
52	makes the tournament return "no item" once only the basket remains — it
53	is the whole basis of the clean `none` loop-exit. Do NOT rely on
54	geometry to reject the basket.
55	- >
56	This subgraph declares NO inputs (`inputs: {}`) and is fully
57	self-contained. Every value that tells the perceive nodes WHAT to look
58	for is a LITERAL string hardcoded in the node — use exactly the node input
59	names `object_name` and `object_description` from
60	examples/canonical_subgraph.json (do not rename them to `item_name`,
61	`container_name`, `item_description`, etc.). DO NOT lift ANY of them to a
62	subgraph input and DO NOT write `Ref("in.<anything>")` for them: there is
63	no upstream producer, so it fails validation (W8: no upstream producer).
64	The basket and item names are known at authoring time — write them in.
65	- >
66	The `decide` router (scripts/<sg>/decide_next_item.py) sits BETWEEN
67	`perceive_item` and `filter_obb_item`, and maps `found -> filter_obb_item`,
68	`none -> none`. Declare `success_values: ["found", "none"]` so BOTH are
69	clean exits; only genuine perception errors fall through to
70	`on_error: perception_failed`.
71	- >
72	TOP-LEVEL ROUTING (do not get this wrong): map this subgraph's
73	`found -> grasp`, `none -> done` (all items packed — a SUCCESS), and
74	`perception_failed -> abort`. `none` is the loop terminator, NOT
75	`perception_failed`; never route `perception_failed` to done.
76	- >
77	Perceive the item from the raw `observe.cameras`, but perceive the
78	container through `exterior_view` (drop the wrist cam) — the angled
79	eye-in-hand view bloats the container OBB by fusing neighbouring points.
80	- >
81	WIRE A SINGLE CHAIN, never parallel branches: observe → exterior_view →
82	perceive_container → filter_obb_container → perceive_item → decide.
83	Do NOT add a second edge from `observe` straight to `perceive_item` to
84	"parallelize" the two perceptions: the runtime scheduler has no join
85	barrier, so `decide` (which consumes `filter_obb_container.obb`) can be
86	scheduled by `perceive_item`'s completion before `filter_obb_container`
87	has run — the $ref fails at runtime and the subgraph aborts on every
88	trial (validator rule S12 rej

People who liked this also viewed

6 picks

Type