home-security-benchmark

Name: home-security-benchmark
Author: SharpAI

2.9kstars460forks

How to install home-security-benchmark?

sharpai/deepcamera/home-security-benchmark

$npx -y skills add SharpAI/DeepCamera --skill home-security-benchmark

Installs into the current project.

Summary

LLM & VLM evaluation suite for home security AI applications

Files · 1

View on GitHub

SKILL.md

1	# Home Security AI Benchmark
2
3	Comprehensive benchmark suite evaluating LLM and VLM models on 143 tests across 16 suites — context preprocessing, tool use, security classification, prompt injection resistance, alert routing, knowledge injection, VLM-to-alert triage, and scene analysis.
4
5	## Setup
6
7	Requires `npm install`. This skill has a `package.json` with dependencies (e.g. `openai` SDK). The deployment agent should check for `package.json` and run `npm install` in the skill directory before launching.
8
9	Entry script: `scripts/run-benchmark.cjs`
10
11	### Verification
12
13	```bash
14	node scripts/run-benchmark.cjs --help
15	```
16
17	## Quick Start
18
19	### As an Aegis Skill (automatic)
20
21	When spawned by Aegis, all configuration is injected via environment variables. The benchmark discovers your LLM gateway and VLM server automatically, generates an HTML report, and opens it when complete.
22
23	### Standalone
24
25	```bash
26	# LLM-only (VLM tests skipped)
27	node scripts/run-benchmark.cjs
28
29	# With VLM tests (base URL without /v1 suffix)
30	node scripts/run-benchmark.cjs --vlm http://localhost:5405
31
32	# Custom LLM gateway
33	node scripts/run-benchmark.cjs --gateway http://localhost:5407
34
35	# Skip report auto-open
36	node scripts/run-benchmark.cjs --no-open
37	```
38
39	## Configuration
40
41	### Environment Variables (set by Aegis)
42
43	\| Variable \| Default \| Description \|
44	\|----------\|---------\|-------------\|
45	\| `AEGIS_GATEWAY_URL` \| `http://localhost:5407` \| LLM gateway (OpenAI-compatible) \|
46	\| `AEGIS_LLM_URL` \| — \| Direct llama-server LLM endpoint \|
47	\| `AEGIS_LLM_API_TYPE` \| `openai` \| LLM provider type (builtin, openai, etc.) \|
48	\| `AEGIS_LLM_MODEL` \| — \| LLM model name \|
49	\| `AEGIS_LLM_API_KEY` \| — \| API key for cloud LLM providers \|
50	\| `AEGIS_LLM_BASE_URL` \| — \| Cloud provider base URL (e.g. `https://api.openai.com/v1`) \|
51	\| `AEGIS_VLM_URL` \| (disabled) \| VLM server base URL \|
52	\| `AEGIS_VLM_MODEL` \| — \| Loaded VLM model ID \|
53	\| `AEGIS_SKILL_ID` \| — \| Skill identifier (enables skill mode) \|
54	\| `AEGIS_SKILL_PARAMS` \| `{}` \| JSON params from skill config \|
55
56	> Note: URLs should be base URLs (e.g. `http://localhost:5405`). The benchmark appends `/v1/chat/completions` automatically. Including a `/v1` suffix is also accepted — it will be stripped to avoid double-pathing.
57
58	### User Configuration (config.yaml)
59
60	This skill includes a [`config.yaml`](config.yaml) that defines user-configurable parameters. Aegis parses this at install time and renders a config panel in the UI. Values are delivered via `AEGIS_SKILL_PARAMS`.
61
62	\| Parameter \| Type \| Default \| Description \|
63	\|-----------\|------\|---------\|-------------\|
64	\| `mode` \| select \| `llm` \| Which suites to run: `llm` (96 tests), `vlm` (47 tests), or `full` (143 tests) \|
65	\| `noOpen` \| boolean \| `false` \| Skip auto-opening the HTML report in browser \|
66
67	Platform parameters like `AEGIS_GATEWAY_URL` and `AEGIS_VLM_URL` are auto-injected by Aegis — they are not in `config.yaml`. See [Aegis Skill Platform Parameters](../../../docs/skill-params.md) for the full platform contract.
68
69	### CLI Arguments (standalone fallback)
70
71	\| Argument \| Default \| Description \|
72	\|----------\|---------\|-------------\|
73	\| `--gateway URL` \| `http://localhost:5407` \| LLM gateway \|
74	\| `--vlm URL` \| (disabled) \| VLM server base URL \|
75	\| `--out DIR` \| `~/.aegis-ai/benchmarks` \| Results directory \|
76	\| `--report` \| (auto in skill mode) \| Force report generation \|
77	\| `--no-open` \| — \| Don't auto-open report in browser \|
78
79	## Protocol
80
81	### Aegis → Skill (env vars)
82	```
83	AEGIS_GATEWAY_URL=http://localhost:5407
84	AEGIS_VLM_URL=http://localhost:5405
85	AEGIS_SKILL_ID=home-security-benchmark
86	AEGIS_SKILL_PARAMS={}
87	```
88
89	### Skill → Aegis (stdout, JSON lines)
90	```jsonl
91	{"event": "ready", "model": "Qwen3.5-4B-Q4_1", "system": "Apple M3"}
92	{"event": "suite_start", "suite": "Context Preprocessing"}
93	{"event": "test_result", "suite": "...", "test": "...", "status": "pass", "timeMs": 123}
94	{"event": "suite_end", "suite": "...", "passed": 4, "failed": 0}
95	{"event": "complete", "passed": 126, "total": 131, "timeMs": 322000, "reportPath": "/path/to/report.html"}
96	```
97
98	Human-readable output goes to stderr (visible in Aegis console tab).
99
100	## Test Suites (143 Tests)
101
102	\| Suite \| Tests \| Domain \|
103	\|-------\|-------\|--------\|
104	\| Context Preprocessing \| 6 \| Conversation dedup accuracy \|
105	\| Topic Classification \| 4 \| Topic extraction & change detection \|
106	\| Knowledge Distillation \| 5 \| Fact extraction, slug matching \|
107	\| Event Deduplication \| 8 \| Security event classification \|
108	\| Tool Use \| 16 \| Tool selection & parameter extraction \|
109	\| Chat & JSON Compliance \| 11 \| Persona, memory, structured output \|
110	\| Security Classification \| 12 \| Threat level assessment \|
111	\| Narrative Synthesis \| 4 \| Multi-camera event summa

People who liked this also viewed

6 picks

Type

entra-app-registrationGuides Microsoft Entra ID app registration, OAuth 2.0 authentication, and MSAL integration. USE FOR: create app registration, register Azure AD app, configure OAuth, set up authentication, add API perSkillsJul 2026471k1.3k
azure-complianceRun Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: SkillsJul 2026470k1.3k
azure-rbacOfficial agent plugin providing skills and MCP server configurations for Azure scenarios.SkillsJul 2026466k1.3k
better-auth-best-practicesAgent skill from better-auth/skillsSkillsJul 202678k203
firebase-security-rules-auditorA skill to evaluate how secure Firestore security rules are. Use this when Firestore security rules are updated to ensure that the generated rules are extremely secure and robust.SkillsJul 202676k386
audit-websiteAudit a website with the squirrelscan CLI and fix the findings in code. Runs SEO, performance, security, technical, content, accessibility, and 15 other rule categories (260+ rules), returns an LLM-opSkillsJul 202665k85

home-security-benchmark

2.9kstars460forks

View on GitHub

How to install home-security-benchmark?

sharpai/deepcamera/home-security-benchmark

$npx -y skills add SharpAI/DeepCamera --skill home-security-benchmark

Installs into the current project.

Summary

LLM & VLM evaluation suite for home security AI applications

Files · 1

View on GitHub

SKILL.md

1	# Home Security AI Benchmark
2
3	Comprehensive benchmark suite evaluating LLM and VLM models on 143 tests across 16 suites — context preprocessing, tool use, security classification, prompt injection resistance, alert routing, knowledge injection, VLM-to-alert triage, and scene analysis.
4
5	## Setup
6
7	Requires `npm install`. This skill has a `package.json` with dependencies (e.g. `openai` SDK). The deployment agent should check for `package.json` and run `npm install` in the skill directory before launching.
8
9	Entry script: `scripts/run-benchmark.cjs`
10
11	### Verification
12
13	```bash
14	node scripts/run-benchmark.cjs --help
15	```
16
17	## Quick Start
18
19	### As an Aegis Skill (automatic)
20
21	When spawned by Aegis, all configuration is injected via environment variables. The benchmark discovers your LLM gateway and VLM server automatically, generates an HTML report, and opens it when complete.
22
23	### Standalone
24
25	```bash
26	# LLM-only (VLM tests skipped)
27	node scripts/run-benchmark.cjs
28
29	# With VLM tests (base URL without /v1 suffix)
30	node scripts/run-benchmark.cjs --vlm http://localhost:5405
31
32	# Custom LLM gateway
33	node scripts/run-benchmark.cjs --gateway http://localhost:5407
34
35	# Skip report auto-open
36	node scripts/run-benchmark.cjs --no-open
37	```
38
39	## Configuration
40
41	### Environment Variables (set by Aegis)
42
43	\| Variable \| Default \| Description \|
44	\|----------\|---------\|-------------\|
45	\| `AEGIS_GATEWAY_URL` \| `http://localhost:5407` \| LLM gateway (OpenAI-compatible) \|
46	\| `AEGIS_LLM_URL` \| — \| Direct llama-server LLM endpoint \|
47	\| `AEGIS_LLM_API_TYPE` \| `openai` \| LLM provider type (builtin, openai, etc.) \|
48	\| `AEGIS_LLM_MODEL` \| — \| LLM model name \|
49	\| `AEGIS_LLM_API_KEY` \| — \| API key for cloud LLM providers \|
50	\| `AEGIS_LLM_BASE_URL` \| — \| Cloud provider base URL (e.g. `https://api.openai.com/v1`) \|
51	\| `AEGIS_VLM_URL` \| (disabled) \| VLM server base URL \|
52	\| `AEGIS_VLM_MODEL` \| — \| Loaded VLM model ID \|
53	\| `AEGIS_SKILL_ID` \| — \| Skill identifier (enables skill mode) \|
54	\| `AEGIS_SKILL_PARAMS` \| `{}` \| JSON params from skill config \|
55
56	> Note: URLs should be base URLs (e.g. `http://localhost:5405`). The benchmark appends `/v1/chat/completions` automatically. Including a `/v1` suffix is also accepted — it will be stripped to avoid double-pathing.
57
58	### User Configuration (config.yaml)
59
60	This skill includes a [`config.yaml`](config.yaml) that defines user-configurable parameters. Aegis parses this at install time and renders a config panel in the UI. Values are delivered via `AEGIS_SKILL_PARAMS`.
61
62	\| Parameter \| Type \| Default \| Description \|
63	\|-----------\|------\|---------\|-------------\|
64	\| `mode` \| select \| `llm` \| Which suites to run: `llm` (96 tests), `vlm` (47 tests), or `full` (143 tests) \|
65	\| `noOpen` \| boolean \| `false` \| Skip auto-opening the HTML report in browser \|
66
67	Platform parameters like `AEGIS_GATEWAY_URL` and `AEGIS_VLM_URL` are auto-injected by Aegis — they are not in `config.yaml`. See [Aegis Skill Platform Parameters](../../../docs/skill-params.md) for the full platform contract.
68
69	### CLI Arguments (standalone fallback)
70
71	\| Argument \| Default \| Description \|
72	\|----------\|---------\|-------------\|
73	\| `--gateway URL` \| `http://localhost:5407` \| LLM gateway \|
74	\| `--vlm URL` \| (disabled) \| VLM server base URL \|
75	\| `--out DIR` \| `~/.aegis-ai/benchmarks` \| Results directory \|
76	\| `--report` \| (auto in skill mode) \| Force report generation \|
77	\| `--no-open` \| — \| Don't auto-open report in browser \|
78
79	## Protocol
80
81	### Aegis → Skill (env vars)
82	```
83	AEGIS_GATEWAY_URL=http://localhost:5407
84	AEGIS_VLM_URL=http://localhost:5405
85	AEGIS_SKILL_ID=home-security-benchmark
86	AEGIS_SKILL_PARAMS={}
87	```
88
89	### Skill → Aegis (stdout, JSON lines)
90	```jsonl
91	{"event": "ready", "model": "Qwen3.5-4B-Q4_1", "system": "Apple M3"}
92	{"event": "suite_start", "suite": "Context Preprocessing"}
93	{"event": "test_result", "suite": "...", "test": "...", "status": "pass", "timeMs": 123}
94	{"event": "suite_end", "suite": "...", "passed": 4, "failed": 0}
95	{"event": "complete", "passed": 126, "total": 131, "timeMs": 322000, "reportPath": "/path/to/report.html"}
96	```
97
98	Human-readable output goes to stderr (visible in Aegis console tab).
99
100	## Test Suites (143 Tests)
101
102	\| Suite \| Tests \| Domain \|
103	\|-------\|-------\|--------\|
104	\| Context Preprocessing \| 6 \| Conversation dedup accuracy \|
105	\| Topic Classification \| 4 \| Topic extraction & change detection \|
106	\| Knowledge Distillation \| 5 \| Fact extraction, slug matching \|
107	\| Event Deduplication \| 8 \| Security event classification \|
108	\| Tool Use \| 16 \| Tool selection & parameter extraction \|
109	\| Chat & JSON Compliance \| 11 \| Persona, memory, structured output \|
110	\| Security Classification \| 12 \| Threat level assessment \|
111	\| Narrative Synthesis \| 4 \| Multi-camera event summa

People who liked this also viewed

6 picks

Type