robot-perception

Name: robot-perception
Author: arpitg1304

320stars44forks

How to install robot-perception?

arpitg1304/robotics-agent-skills/robot-perception

$npx -y skills add arpitg1304/robotics-agent-skills --skill robot-perception

Installs into the current project.

Summary

Comprehensive best practices for robot perception systems covering cameras, LiDARs, depth sensors, IMUs, and multi-sensor setups. Use this skill when working with RGB image processing, depth maps, point clouds, sensor calibration (intrinsic, extrinsic, hand-eye), object detection

Files · 1

View on GitHub

SKILL.md

1	# Robot Perception Skill
2
3	## When to Use This Skill
4	- Setting up and configuring camera, LiDAR, or depth sensors
5	- Building RGB, depth, or point cloud processing pipelines
6	- Calibrating cameras (intrinsic, extrinsic, hand-eye)
7	- Implementing object detection, segmentation, or tracking for robots
8	- Fusing data from multiple sensor modalities
9	- Streaming sensor data with proper threading and buffering
10	- Synchronizing multi-sensor rigs
11	- Deploying perception models on robot hardware (GPU, edge)
12	- Debugging perception failures (latency, dropped frames, misalignment)
13
14	## Sensor Landscape
15
16	### Sensor Types and Characteristics
17
18	```
19	Sensor Type Output Range Rate Best For
20	─────────────────────────────────────────────────────────────────────────
21	RGB Camera (H,W,3) uint8 ∞ 30-120Hz Object detection, tracking, visual servoing
22	Stereo Camera (H,W,3)+(H,W,3) 0.3-20m 30-90Hz Dense depth from passive stereo
23	Structured Light (H,W) float + RGB 0.2-10m 30Hz Indoor manipulation, short range
24	ToF Depth (H,W) float + RGB 0.1-10m 30Hz Indoor, medium range
25	LiDAR (spinning) (N,3) or (N,4) 0.5-200m 10-20Hz Outdoor navigation, mapping
26	LiDAR (solid-st.) (N,3) 0.5-200m 10-30Hz Automotive, outdoor
27	IMU (6,) or (9,) N/A 200-1kHz Orientation, motion estimation
28	Force/Torque (6,) float N/A 1kHz+ Contact detection, force control
29	Tactile (H,W) or (N,3) Contact 30-100Hz Grasp quality, texture
30	Event Camera Events (x,y,t,p) ∞ μs High-speed tracking, HDR scenes
31	```
32
33	### Common Sensor Hardware
34
35	```
36	Device Type SDK/Driver ROS2 Package
37	──────────────────────────────────────────────────────────────────────────
38	Intel RealSense Structured Light pyrealsense2 realsense2_camera
39	Stereolabs ZED Stereo + IMU pyzed zed_wrapper
40	Luxonis OAK-D Stereo + Neural depthai depthai_ros
41	FLIR/Basler Industrial RGB PySpin/pypylon spinnaker_camera_driver
42	Velodyne Spinning LiDAR velodyne_driver velodyne
43	Ouster Spinning LiDAR ouster-sdk ros2_ouster
44	Livox Solid-state LiDAR livox_sdk livox_ros2_driver
45	USB Webcam RGB OpenCV VideoCapture usb_cam / v4l2_camera
46	```
47
48	## Camera Models and Calibration
49
50	### Pinhole Camera Model
51
52	```
53	3D World Point (X, Y, Z)
54	\|
55	[R \| t] — Extrinsic (world → camera)
56	\|
57	Camera Point (Xc, Yc, Zc)
58	\|
59	K — Intrinsic (camera → pixel)
60	\|
61	Pixel (u, v)
62
63	K = [ fx 0 cx ] fx, fy = focal lengths (pixels)
64	[ 0 fy cy ] cx, cy = principal point
65	[ 0 0 1 ]
66
67	Projection: [u, v, 1]^T = K @ [R \| t] @ [X, Y, Z, 1]^T
68	```
69
70	### Intrinsic Calibration
71
72	```python
73	import cv2
74	import numpy as np
75	from pathlib import Path
76
77	class IntrinsicCalibrator:
78	"""Camera intrinsic calibration using checkerboard pattern"""
79
80	def __init__(self, board_size=(9, 6), square_size_m=0.025):
81	self.board_size = board_size
82	self.square_size = square_size_m
83
84	# Prepare object points (3D coordinates of checkerboard corners)
85	self.objp = np.zeros((board_size[0] * board_size[1], 3), np.float32)
86	self.objp[:, :2] = np.mgrid[
87	0:board_size[0], 0:board_size[1]
88	].T.reshape(-1, 2) * square_size_m
89
90	def collect_calibration_images(self, camera, num_images=30,
91	min_coverage=0.6):
92	"""Collect calibration images with good spatial coverage.
93
94	IMPORTANT: Move the board to cover all regions of the image,
95	including corners and edges. Tilt the board at various angles.
96	Bad coverage = bad calibration, especially a

People who liked this also viewed

6 picks

Type

robot-perception

320stars44forks

View on GitHub

How to install robot-perception?

arpitg1304/robotics-agent-skills/robot-perception

$npx -y skills add arpitg1304/robotics-agent-skills --skill robot-perception

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Robot Perception Skill
2
3	## When to Use This Skill
4	- Setting up and configuring camera, LiDAR, or depth sensors
5	- Building RGB, depth, or point cloud processing pipelines
6	- Calibrating cameras (intrinsic, extrinsic, hand-eye)
7	- Implementing object detection, segmentation, or tracking for robots
8	- Fusing data from multiple sensor modalities
9	- Streaming sensor data with proper threading and buffering
10	- Synchronizing multi-sensor rigs
11	- Deploying perception models on robot hardware (GPU, edge)
12	- Debugging perception failures (latency, dropped frames, misalignment)
13
14	## Sensor Landscape
15
16	### Sensor Types and Characteristics
17
18	```
19	Sensor Type Output Range Rate Best For
20	─────────────────────────────────────────────────────────────────────────
21	RGB Camera (H,W,3) uint8 ∞ 30-120Hz Object detection, tracking, visual servoing
22	Stereo Camera (H,W,3)+(H,W,3) 0.3-20m 30-90Hz Dense depth from passive stereo
23	Structured Light (H,W) float + RGB 0.2-10m 30Hz Indoor manipulation, short range
24	ToF Depth (H,W) float + RGB 0.1-10m 30Hz Indoor, medium range
25	LiDAR (spinning) (N,3) or (N,4) 0.5-200m 10-20Hz Outdoor navigation, mapping
26	LiDAR (solid-st.) (N,3) 0.5-200m 10-30Hz Automotive, outdoor
27	IMU (6,) or (9,) N/A 200-1kHz Orientation, motion estimation
28	Force/Torque (6,) float N/A 1kHz+ Contact detection, force control
29	Tactile (H,W) or (N,3) Contact 30-100Hz Grasp quality, texture
30	Event Camera Events (x,y,t,p) ∞ μs High-speed tracking, HDR scenes
31	```
32
33	### Common Sensor Hardware
34
35	```
36	Device Type SDK/Driver ROS2 Package
37	──────────────────────────────────────────────────────────────────────────
38	Intel RealSense Structured Light pyrealsense2 realsense2_camera
39	Stereolabs ZED Stereo + IMU pyzed zed_wrapper
40	Luxonis OAK-D Stereo + Neural depthai depthai_ros
41	FLIR/Basler Industrial RGB PySpin/pypylon spinnaker_camera_driver
42	Velodyne Spinning LiDAR velodyne_driver velodyne
43	Ouster Spinning LiDAR ouster-sdk ros2_ouster
44	Livox Solid-state LiDAR livox_sdk livox_ros2_driver
45	USB Webcam RGB OpenCV VideoCapture usb_cam / v4l2_camera
46	```
47
48	## Camera Models and Calibration
49
50	### Pinhole Camera Model
51
52	```
53	3D World Point (X, Y, Z)
54	\|
55	[R \| t] — Extrinsic (world → camera)
56	\|
57	Camera Point (Xc, Yc, Zc)
58	\|
59	K — Intrinsic (camera → pixel)
60	\|
61	Pixel (u, v)
62
63	K = [ fx 0 cx ] fx, fy = focal lengths (pixels)
64	[ 0 fy cy ] cx, cy = principal point
65	[ 0 0 1 ]
66
67	Projection: [u, v, 1]^T = K @ [R \| t] @ [X, Y, Z, 1]^T
68	```
69
70	### Intrinsic Calibration
71
72	```python
73	import cv2
74	import numpy as np
75	from pathlib import Path
76
77	class IntrinsicCalibrator:
78	"""Camera intrinsic calibration using checkerboard pattern"""
79
80	def __init__(self, board_size=(9, 6), square_size_m=0.025):
81	self.board_size = board_size
82	self.square_size = square_size_m
83
84	# Prepare object points (3D coordinates of checkerboard corners)
85	self.objp = np.zeros((board_size[0] * board_size[1], 3), np.float32)
86	self.objp[:, :2] = np.mgrid[
87	0:board_size[0], 0:board_size[1]
88	].T.reshape(-1, 2) * square_size_m
89
90	def collect_calibration_images(self, camera, num_images=30,
91	min_coverage=0.6):
92	"""Collect calibration images with good spatial coverage.
93
94	IMPORTANT: Move the board to cover all regions of the image,
95	including corners and edges. Tilt the board at various angles.
96	Bad coverage = bad calibration, especially a

People who liked this also viewed

6 picks

Type