homesafe-bench
starsforks
How to install homesafe-bench?
sharpai/deepcamera/homesafe-bench
$npx -y skills add SharpAI/DeepCamera --skill homesafe-benchVLM indoor safety hazard detection benchmark inspired by HomeSafeBench (arXiv 2509.23690)
SKILL.md
| 1 | # HomeSafe-Bench — Indoor Safety Hazard Detection |
| 2 | |
| 3 | VLM benchmark evaluating indoor home safety hazard detection from static camera frames. Inspired by [HomeSafeBench (arXiv 2509.23690)](https://arxiv.org/abs/2509.23690), adapted for **fixed indoor security cameras** with AI-generated fixture images. |
| 4 | |
| 5 | **40 tests** across **5 hazard categories**: |
| 6 | |
| 7 | | Category | Tests | Example Hazards | |
| 8 | |----------|-------|-----------------| |
| 9 | | 🔥 Fire / Smoke | 8 | Stove smoke, candle near curtain, iron left face-down | |
| 10 | | ⚡ Electrical | 8 | Overloaded power strip, exposed wire, frayed cord | |
| 11 | | 🚶 Trip / Fall | 8 | Toys on staircase, wet floor, loose rug edge | |
| 12 | | 👶 Child Safety | 8 | Open cabinet with chemicals, uncovered outlet, blind cord | |
| 13 | | 📦 Falling Objects | 8 | Heavy items on high shelf, leaning bookshelf, loose ceiling fan | |
| 14 | |
| 15 | ## Setup |
| 16 | |
| 17 | **Requires `npm install`.** This skill has a `package.json` with dependencies (e.g. `openai` SDK). The deployment agent should check for `package.json` and run `npm install` in the skill directory before launching. |
| 18 | |
| 19 | Entry script: `scripts/run-benchmark.cjs` |
| 20 | |
| 21 | ### Verification |
| 22 | |
| 23 | ```bash |
| 24 | node scripts/run-benchmark.cjs --help |
| 25 | ``` |
| 26 | |
| 27 | ## Quick Start |
| 28 | |
| 29 | ### As an Aegis Skill (automatic) |
| 30 | |
| 31 | When spawned by Aegis, all configuration is injected via environment variables. The benchmark discovers your VLM server automatically, generates an HTML report, and opens it when complete. |
| 32 | |
| 33 | ### Standalone |
| 34 | |
| 35 | ```bash |
| 36 | # Run all 40 tests |
| 37 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 |
| 38 | |
| 39 | # Quick mode (2 tests per category = 10 total) |
| 40 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --mode quick |
| 41 | |
| 42 | # Skip report auto-open |
| 43 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --no-open |
| 44 | ``` |
| 45 | |
| 46 | ## Configuration |
| 47 | |
| 48 | ### Environment Variables (set by Aegis) |
| 49 | |
| 50 | | Variable | Default | Description | |
| 51 | |----------|---------|-------------| |
| 52 | | `AEGIS_VLM_URL` | *(required)* | VLM server base URL | |
| 53 | | `AEGIS_VLM_MODEL` | — | Loaded VLM model ID | |
| 54 | | `AEGIS_SKILL_ID` | — | Skill identifier (enables skill mode) | |
| 55 | | `AEGIS_SKILL_PARAMS` | `{}` | JSON params from skill config | |
| 56 | |
| 57 | > **Note**: URLs should be base URLs (e.g. `http://localhost:5405`). The benchmark appends `/v1/chat/completions` automatically. |
| 58 | |
| 59 | ### User Configuration (config.yaml) |
| 60 | |
| 61 | | Parameter | Type | Default | Description | |
| 62 | |-----------|------|---------|-------------| |
| 63 | | `mode` | select | `full` | Which mode: `full` (40 tests) or `quick` (10 tests — 2 per category) | |
| 64 | | `noOpen` | boolean | `false` | Skip auto-opening the HTML report in browser | |
| 65 | |
| 66 | ### CLI Arguments (standalone fallback) |
| 67 | |
| 68 | | Argument | Default | Description | |
| 69 | |----------|---------|-------------| |
| 70 | | `--vlm URL` | *(required)* | VLM server base URL | |
| 71 | | `--mode MODE` | `full` | Test mode: `full` or `quick` | |
| 72 | | `--out DIR` | `~/.aegis-ai/homesafe-benchmarks` | Results directory | |
| 73 | | `--no-open` | — | Don't auto-open report in browser | |
| 74 | |
| 75 | ## Protocol |
| 76 | |
| 77 | ### Aegis → Skill (env vars) |
| 78 | ``` |
| 79 | AEGIS_VLM_URL=http://localhost:5405 |
| 80 | AEGIS_SKILL_ID=homesafe-bench |
| 81 | AEGIS_SKILL_PARAMS={} |
| 82 | ``` |
| 83 | |
| 84 | ### Skill → Aegis (stdout, JSON lines) |
| 85 | ```jsonl |
| 86 | {"event": "ready", "vlm": "SmolVLM-500M", "system": "Apple M3"} |
| 87 | {"event": "suite_start", "suite": "🔥 Fire / Smoke"} |
| 88 | {"event": "test_result", "suite": "...", "test": "...", "status": "pass", "timeMs": 4500} |
| 89 | {"event": "suite_end", "suite": "...", "passed": 7, "failed": 1} |
| 90 | {"event": "complete", "passed": 36, "total": 40, "timeMs": 180000, "reportPath": "/path/to/report.html"} |
| 91 | ``` |
| 92 | |
| 93 | Human-readable output goes to **stderr** (visible in Aegis console tab). |
| 94 | |
| 95 | ## Citation |
| 96 | |
| 97 | This benchmark is inspired by: |
| 98 | |
| 99 | > **HomeSafeBench: Towards Measuring the Proficiency of Home Safety for Embodied AI Agents** |
| 100 | > arXiv:2509.23690 |
| 101 | > |
| 102 | > Unlike the academic benchmark (embodied agent + navigation in simulated 3D environments), our version uses **static indoor camera frames** — matching real-world indoor security camera deployment (fixed wall/ceiling mount). All fixture images are **AI-generated** consistent with DeepCamera's privacy-first approach. |
| 103 | |
| 104 | ## Requirements |
| 105 | |
| 106 | - Node.js ≥ 18 |
| 107 | - `npm install` (for `openai` SDK dependency) |
| 108 | - Running VLM server (llama-server with vision model, or OpenAI-compatible VLM endpoint) |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
