agentsight-testing

Name: agentsight-testing
Author: eunomia-bpf

534stars72forks

How to install agentsight-testing?

eunomia-bpf/agentsight/agentsight-testing

$npx -y skills add eunomia-bpf/agentsight --skill agentsight-testing

Installs into the current project.

Summary

Validate AgentSight changes before PR completion. Use when Codex changes AgentSight capture, parsing, top/TUI, web UI, reporting, agent-native session handling, CI scripts, or release-sensitive behavior and must run automated tests plus real CLI/TUI/UI/agent smoke checks.

Files · 1

View on GitHub

SKILL.md

1	# AgentSight Testing
2
3	## Goal
4
5	Prove that an AgentSight change works at the product boundary, not only at the
6	unit-test boundary. Prefer real binaries, real commands, and saved evidence.
7	Mocks are allowed only for narrow regression tests; they do not replace real
8	agent validation when the relevant CLI is installed and authenticated.
9
10	CI can supplement this proof, but it does not replace direct observation. The
11	agent doing the change must inspect the TUI/UI/CLI output or saved artifacts
12	itself and report what was observed. Do not defer missing observation to "the
13	next PR" or say CI should cover it unless the user explicitly narrows the task
14	to CI work.
15
16	## Validation Ladder
17
18	Run the narrowest relevant check first, then widen until the changed behavior
19	has been exercised end to end.
20
21	1. Unit/regression tests:
22	- `cargo test --manifest-path collector/Cargo.toml <test_name> -- --nocapture`
23	- `cargo test --manifest-path agent-session/Cargo.toml <test_name> -- --nocapture`
24	2. Component suites:
25	- `cargo test --manifest-path collector/Cargo.toml`
26	- `cargo test --manifest-path agent-session/Cargo.toml`
27	- `cd frontend && npm run lint && npm run build`
28	3. CLI command smoke:
29	- `collector/target/debug/agentsight top --plain --once`
30	- `collector/target/debug/agentsight report list`
31	- `collector/target/debug/agentsight report summary`
32	- `collector/target/debug/agentsight report token`
33	- `collector/target/debug/agentsight stat -- <short command>` when the change affects stat/record paths
34	4. TUI smoke:
35	- Run `agentsight top` in a real terminal or PTY.
36	- Capture the screen text or screenshot artifact. Prefer terminal buffer
37	capture such as `tmux capture-pane`, `script` output that includes the
38	rendered screen, or a real screenshot; raw alt-screen escape logs without
39	readable fields are not sufficient.
40	- Verify key fields by screen buffer/text, not OCR: session id, agent, state, tokens, last message, model, evidence.
41	5. Web UI smoke:
42	- Start a command that serves the web UI (`record`, `stat`, or the relevant server mode).
43	- Open the URL in a browser or Playwright.
44	- Verify the page loads real session/event data and save a screenshot artifact.
45	- The screenshot/artifact must show the relevant feature state, not only that
46	a page rendered or that some JSON endpoint returned data.
47	6. Real supported-agent smoke:
48	- Check installed/authenticated CLIs with `command -v`.
49	- Native session agents: `claude`, `codex`, `gemini`.
50	- Live/record targets from README support: Claude Code, Codex, Gemini CLI, OpenCode, OpenClaw/Node/OpenAI-compatible clients, and any command path touched by the change.
51	- For each available CLI, run a short non-destructive prompt under AgentSight and verify the expected rows/events. Do not count a mock agent as satisfying this step.
52	- Seeing generic captured data is not enough. Verify the product-level fields
53	affected by the change, such as prompt text, response/last message, token
54	counts, model, session binding, tool calls, evidence, and report/top rows.
55
56	## Real Agent Rules
57
58	- Use short prompts that do not modify the repository unless the test is
59	specifically about file effects. Prefer "print a short JSON object" or
60	"answer with one word" prompts.
61	- Record the exact command, agent version when available, exit status, DB path
62	or output artifact, and the AgentSight command used to inspect it.
63	- If a CLI is not installed, not authenticated, rate-limited, or requires
64	unavailable credentials, report it as a blocker or explicit unverified gap.
65	Do not silently skip it.
66	- If one supported agent fails and the failure is in AgentSight, fix it before
67	completing the PR. If the failure is external, capture enough evidence to
68	distinguish external failure from AgentSight failure.
69
70	## Evidence Contract
71
72	Before saying a PR is done, provide:
73
74	- Commands run and pass/fail status.
75	- TUI screen text or screenshot path, plus the specific fields observed.
76	- Web UI screenshot path or Playwright artifact path, plus the specific UI state observed.
77	- Real agent matrix: agent, installed/authenticated status, AgentSight command,
78	observed output, DB/artifact path when available, inspection command, fields
79	verified, and result.
80	- CI links after push.
81	- Copilot review/comment/thread status for the final pushed head.
82
83	## Guardrails
84
85	- Do not relax CI to make validation pass.
86	- Do not replace a required real-agent test with a fixture unless the real CLI
87	is unavailable and that gap is reported.
88	- Keep generated DBs, screenshots, logs, and temporary homes out of commits
89	unless they are intentional fixtures.
90	- Avoid expo

People who liked this also viewed

6 picks

Type

agentsight-testing

534stars72forks

View on GitHub

How to install agentsight-testing?

eunomia-bpf/agentsight/agentsight-testing

$npx -y skills add eunomia-bpf/agentsight --skill agentsight-testing

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AgentSight Testing
2
3	## Goal
4
5	Prove that an AgentSight change works at the product boundary, not only at the
6	unit-test boundary. Prefer real binaries, real commands, and saved evidence.
7	Mocks are allowed only for narrow regression tests; they do not replace real
8	agent validation when the relevant CLI is installed and authenticated.
9
10	CI can supplement this proof, but it does not replace direct observation. The
11	agent doing the change must inspect the TUI/UI/CLI output or saved artifacts
12	itself and report what was observed. Do not defer missing observation to "the
13	next PR" or say CI should cover it unless the user explicitly narrows the task
14	to CI work.
15
16	## Validation Ladder
17
18	Run the narrowest relevant check first, then widen until the changed behavior
19	has been exercised end to end.
20
21	1. Unit/regression tests:
22	- `cargo test --manifest-path collector/Cargo.toml <test_name> -- --nocapture`
23	- `cargo test --manifest-path agent-session/Cargo.toml <test_name> -- --nocapture`
24	2. Component suites:
25	- `cargo test --manifest-path collector/Cargo.toml`
26	- `cargo test --manifest-path agent-session/Cargo.toml`
27	- `cd frontend && npm run lint && npm run build`
28	3. CLI command smoke:
29	- `collector/target/debug/agentsight top --plain --once`
30	- `collector/target/debug/agentsight report list`
31	- `collector/target/debug/agentsight report summary`
32	- `collector/target/debug/agentsight report token`
33	- `collector/target/debug/agentsight stat -- <short command>` when the change affects stat/record paths
34	4. TUI smoke:
35	- Run `agentsight top` in a real terminal or PTY.
36	- Capture the screen text or screenshot artifact. Prefer terminal buffer
37	capture such as `tmux capture-pane`, `script` output that includes the
38	rendered screen, or a real screenshot; raw alt-screen escape logs without
39	readable fields are not sufficient.
40	- Verify key fields by screen buffer/text, not OCR: session id, agent, state, tokens, last message, model, evidence.
41	5. Web UI smoke:
42	- Start a command that serves the web UI (`record`, `stat`, or the relevant server mode).
43	- Open the URL in a browser or Playwright.
44	- Verify the page loads real session/event data and save a screenshot artifact.
45	- The screenshot/artifact must show the relevant feature state, not only that
46	a page rendered or that some JSON endpoint returned data.
47	6. Real supported-agent smoke:
48	- Check installed/authenticated CLIs with `command -v`.
49	- Native session agents: `claude`, `codex`, `gemini`.
50	- Live/record targets from README support: Claude Code, Codex, Gemini CLI, OpenCode, OpenClaw/Node/OpenAI-compatible clients, and any command path touched by the change.
51	- For each available CLI, run a short non-destructive prompt under AgentSight and verify the expected rows/events. Do not count a mock agent as satisfying this step.
52	- Seeing generic captured data is not enough. Verify the product-level fields
53	affected by the change, such as prompt text, response/last message, token
54	counts, model, session binding, tool calls, evidence, and report/top rows.
55
56	## Real Agent Rules
57
58	- Use short prompts that do not modify the repository unless the test is
59	specifically about file effects. Prefer "print a short JSON object" or
60	"answer with one word" prompts.
61	- Record the exact command, agent version when available, exit status, DB path
62	or output artifact, and the AgentSight command used to inspect it.
63	- If a CLI is not installed, not authenticated, rate-limited, or requires
64	unavailable credentials, report it as a blocker or explicit unverified gap.
65	Do not silently skip it.
66	- If one supported agent fails and the failure is in AgentSight, fix it before
67	completing the PR. If the failure is external, capture enough evidence to
68	distinguish external failure from AgentSight failure.
69
70	## Evidence Contract
71
72	Before saying a PR is done, provide:
73
74	- Commands run and pass/fail status.
75	- TUI screen text or screenshot path, plus the specific fields observed.
76	- Web UI screenshot path or Playwright artifact path, plus the specific UI state observed.
77	- Real agent matrix: agent, installed/authenticated status, AgentSight command,
78	observed output, DB/artifact path when available, inspection command, fields
79	verified, and result.
80	- CI links after push.
81	- Copilot review/comment/thread status for the final pushed head.
82
83	## Guardrails
84
85	- Do not relax CI to make validation pass.
86	- Do not replace a required real-agent test with a fixture unless the real CLI
87	is unavailable and that gap is reported.
88	- Keep generated DBs, screenshots, logs, and temporary homes out of commits
89	unless they are intentional fixtures.
90	- Avoid expo

People who liked this also viewed

6 picks

Type