smarthome-bench
starsforks
How to install smarthome-bench?
sharpai/deepcamera/smarthome-bench
$npx -y skills add SharpAI/DeepCamera --skill smarthome-benchVLM evaluation suite for video anomaly detection in smart home camera footage
SKILL.md
| 1 | # SmartHome Video Anomaly Benchmark |
| 2 | |
| 3 | Benchmark suite evaluating VLM models on **video anomaly detection** across **7 smart home categories** — based on the [SmartHome-Bench](https://github.com/Xinyi-0724/SmartHome-Bench-LLM) dataset (1,203 YouTube smart home camera clips). |
| 4 | |
| 5 | > **Key differentiator from HomeSec-Bench**: This benchmark requires **video understanding** (multi-frame sampling), not single-frame analysis. |
| 6 | |
| 7 | ## Setup |
| 8 | |
| 9 | **Requires `npm install` + system dependencies.** The deployment agent should: |
| 10 | 1. Check for `yt-dlp` and `ffmpeg` in PATH |
| 11 | 2. Run `npm install` in the skill directory |
| 12 | |
| 13 | Entry script: `scripts/run-benchmark.cjs` |
| 14 | |
| 15 | ### Verification |
| 16 | |
| 17 | ```bash |
| 18 | node scripts/run-benchmark.cjs --help |
| 19 | ``` |
| 20 | |
| 21 | ## Quick Start |
| 22 | |
| 23 | ### As an Aegis Skill (automatic) |
| 24 | |
| 25 | When spawned by Aegis, configuration is injected via environment variables. The benchmark downloads video clips, samples frames, evaluates with VLM, and generates an HTML report. |
| 26 | |
| 27 | ### Standalone |
| 28 | |
| 29 | ```bash |
| 30 | # Run with local VLM (subset mode, 50 videos) |
| 31 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 |
| 32 | |
| 33 | # Quick test with 10 videos |
| 34 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --max-videos 10 |
| 35 | |
| 36 | # Full benchmark (all curated clips) |
| 37 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --mode full |
| 38 | |
| 39 | # Filter by category |
| 40 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --categories "Wildlife,Security" |
| 41 | |
| 42 | # Skip download (re-evaluate cached videos) |
| 43 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --skip-download |
| 44 | |
| 45 | # Skip report auto-open |
| 46 | node scripts/run-benchmark.cjs --vlm http://localhost:5405 --no-open |
| 47 | ``` |
| 48 | |
| 49 | ## Configuration |
| 50 | |
| 51 | ### Environment Variables (set by Aegis) |
| 52 | |
| 53 | | Variable | Default | Description | |
| 54 | |----------|---------|-------------| |
| 55 | | `AEGIS_VLM_URL` | *(required)* | VLM server base URL | |
| 56 | | `AEGIS_VLM_MODEL` | — | Loaded VLM model ID | |
| 57 | | `AEGIS_SKILL_ID` | — | Skill identifier (enables skill mode) | |
| 58 | | `AEGIS_SKILL_PARAMS` | `{}` | JSON params from skill config | |
| 59 | |
| 60 | > **Note**: This is a VLM-only benchmark. An LLM gateway is not required. |
| 61 | |
| 62 | ### User Configuration (config.yaml) |
| 63 | |
| 64 | This skill includes a [`config.yaml`](config.yaml) that defines user-configurable parameters. Aegis parses this at install time and renders a config panel in the UI. Values are delivered via `AEGIS_SKILL_PARAMS`. |
| 65 | |
| 66 | | Parameter | Type | Default | Description | |
| 67 | |-----------|------|---------|-------------| |
| 68 | | `mode` | select | `subset` | Which clips to evaluate: `subset` (~50 clips) or `full` (all ~105 curated clips) | |
| 69 | | `maxVideos` | number | `50` | Maximum number of videos to evaluate | |
| 70 | | `categories` | text | `all` | Comma-separated category filter (e.g. `Wildlife,Security`) | |
| 71 | | `noOpen` | boolean | `false` | Skip auto-opening the HTML report in browser | |
| 72 | |
| 73 | ### CLI Arguments (standalone fallback) |
| 74 | |
| 75 | | Argument | Default | Description | |
| 76 | |----------|---------|-------------| |
| 77 | | `--vlm URL` | *(required)* | VLM server base URL | |
| 78 | | `--out DIR` | `~/.aegis-ai/smarthome-bench` | Results directory | |
| 79 | | `--max-videos N` | `50` | Max videos to evaluate | |
| 80 | | `--mode MODE` | `subset` | `subset` or `full` | |
| 81 | | `--categories LIST` | `all` | Comma-separated category filter | |
| 82 | | `--skip-download` | — | Skip video download, use cached | |
| 83 | | `--no-open` | — | Don't auto-open report in browser | |
| 84 | | `--report` | *(auto in skill mode)* | Force report generation | |
| 85 | |
| 86 | ## Protocol |
| 87 | |
| 88 | ### Aegis → Skill (env vars) |
| 89 | ``` |
| 90 | AEGIS_VLM_URL=http://localhost:5405 |
| 91 | AEGIS_SKILL_ID=smarthome-bench |
| 92 | AEGIS_SKILL_PARAMS={} |
| 93 | ``` |
| 94 | |
| 95 | ### Skill → Aegis (stdout, JSON lines) |
| 96 | ```jsonl |
| 97 | {"event": "ready", "model": "SmolVLM2-2.2B", "system": "Apple M3"} |
| 98 | {"event": "suite_start", "suite": "Wildlife"} |
| 99 | {"event": "test_result", "suite": "Wildlife", "test": "smartbench_0003", "status": "pass", "timeMs": 4500} |
| 100 | {"event": "suite_end", "suite": "Wildlife", "passed": 12, "failed": 3} |
| 101 | {"event": "complete", "passed": 78, "total": 105, "timeMs": 480000, "reportPath": "/path/to/report.html"} |
| 102 | ``` |
| 103 | |
| 104 | Human-readable output goes to **stderr** (visible in Aegis console tab). |
| 105 | |
| 106 | ## Test Suites (7 Categories) |
| 107 | |
| 108 | | Suite | Description | Anomaly Examples | |
| 109 | |-------|-------------|------------------| |
| 110 | | 🦊 Wildlife | Wild animals near home cameras | Bear on porch, deer in garden, coyote at night | |
| 111 | | 👴 Senior Care | Elderly activity monitoring | Falls, wandering, unusual inactivity | |
| 112 | | 👶 Baby Monitoring | Infant/child safety | Stroller rolling, child climbing, unsupervised | |
| 113 | | 🐾 Pet Monitoring | Pet behavior detection | Pet illness, escaped pets, unusual behavior | |
| 114 | | 🔒 Home Security | Intrusion & suspicious activity | Break-ins, trespassing, porch pirates | |
| 115 | | 📦 Package Delivery | Package arrival & th |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
