agent-activity-audit

Name: agent-activity-audit
Author: hyperb1iss

45stars8forks

How to install agent-activity-audit?

hyperb1iss/sibyl/agent-activity-audit

$npx -y skills add hyperb1iss/sibyl --skill agent-activity-audit

Installs into the current project.

Summary

Audit recent agent transcripts (Claude Code and Codex) to learn how a tool, system, or skill is actually being used in the wild. Surfaces failure modes, friction, success patterns, and concrete improvement candidates from real session data. Use this when you want to improve a dev

Files · 1

View on GitHub

SKILL.md

1	# Agent Activity Audit
2
3	This skill executes a structured pass over recent agent transcripts to learn what's working and
4	what's hurting. The original audit (May 2026) examined ~30 days of Claude Code and Codex sessions to
5	improve Sibyl itself — see `EXAMPLES.md` for the full reproducible run.
6
7	The output is a synthesis report grounded in real session evidence, plus per-group findings files
8	you can act on directly.
9
10	---
11
12	## When to use
13
14	- You maintain a system that agents call (CLI, MCP server, library, skill) and want signal beyond
15	"did it work?"
16	- You suspect agents are stumbling on something but can't name what.
17	- A planning cycle is about to start and you want product priorities grounded in usage data, not
18	vibes.
19	- A new release shipped and you want to see how it landed in the wild.
20
21	Not for: general code review, security audits, performance benchmarking. This skill reads
22	session transcripts; it doesn't analyze code.
23
24	---
25
26	## Agent rules (READ FIRST)
27
28	1. Always write artifacts under `contexts/<analysis-name>-<date>/`. Keep raw scans, episode
29	extracts, and findings in one tree so the analysis is reproducible and the user can replay or
30	extend it.
31
32	2. Filter early, filter hard. Most transcripts are noise. Triage with cheap grep before spinning
33	up parallel subagents — the goal is to give each subagent ~50-100 KB of focused episode data, not
34	raw multi-MB JSONLs.
35
36	3. Partition by date for the swarm. Date-based partitions are mutually exclusive, cover the full
37	window, and make convergence across groups easy to spot (same theme in 4+ date ranges = durable
38	issue).
39
40	4. Each subagent writes findings to a file. Don't let agents return giant prose back to the main
41	thread. Their job: produce `findings/group_<X>.md`, return a ≤250-word summary.
42
43	5. Convergence-first synthesis. A pain point in 4+ groups is durable. Single-group findings
44	warrant a sanity check before they're elevated. Count evidence; don't trust impressions.
45
46	6. Verify before recommending fixes. Inspect current source for the surfaces the audit
47	implicates. A finding like "the CLI rejects `--kind gotcha`" should point at the enum's actual
48	location.
49
50	7. Capture durable learnings to Sibyl after synthesis. The point is to feed back into the
51	product graph; use `sibyl remember --kind pattern` (or `--kind decision`) on the substantive
52	findings.
53
54	---
55
56	## The Workflow
57
58	```
59	inventory → triage → extract episodes → parallel swarm → synthesis → capture
60	```
61
62	Each step has fall-back behavior if data shape varies between Claude and Codex transcripts.
63
64	### Step 1: Inventory
65
66	Find all JSONLs in the target window. Claude lives in
67	`~/.claude/projects/<project-slug>/<uuid>.jsonl`, Codex lives in
68	`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`. Filter by mtime.
69
70	```bash
71	mkdir -p contexts/<name>-$(date +%F)/triage
72	cd contexts/<name>-$(date +%F)
73
74	# Last 30 days
75	CUTOFF=$(date -v-30d +%F 2>/dev/null \|\| date -d '30 days ago' +%F) # BSD (macOS) or GNU date
76	find ~/.claude/projects -name '*.jsonl' -newermt "$CUTOFF" > triage/claude_files.txt
77	find ~/.codex/sessions -name '*.jsonl' -newermt "$CUTOFF" > triage/codex_files.txt
78	cat triage/claude_files.txt triage/codex_files.txt > triage/all_files.txt
79
80	wc -l triage/*.txt
81	```
82
83	### Step 2: Triage with the scanner
84
85	Use `scripts/scan.py` (ships with this skill) to extract per-file statistics: total events, target
86	tool-call counts, error counts, user corrections. Runs in parallel.
87
88	```bash
89	SKILL_DIR=$(dirname "$(realpath "$0")") # or hard-code the path
90	cat triage/all_files.txt \| xargs -P 12 -n 5 python3 "$SKILL_DIR/scripts/scan.py" \
91	--target <tool-or-skill-keyword> > triage/scan_results.jsonl
92	```
93
94	The scanner detects three shapes of usage:
95
96	- CLI: Bash/exec_command calls whose command line matches the target's CLI name
97	- MCP: tool names matching `mcp__<target>__` or `<target>_`
98	- Skill: Skill invocations whose name matches the target
99
100	Look at the output to confirm signal quality before proceeding.
101
102	### Step 3: Extract focused episodes
103
104	Each "episode" is a target-tool call plus its preceding user message, assistant text, and tool
105	result. Use `scripts/extract_episodes.py` to write per-session markdown files (~10-30 KB each, vs
106	the 50-500 KB raw transcripts).
107
108	```bash
109	# Filter to files with actual usage
110	python3 -c "
111	import json
112	for line in open('triage/scan_results.jsonl'):
113	r = json.loads(line)
114	if r.get('target_total', 0) > 0:
115	print(r['path'])
116	" > triage/using_files.txt
117
118	mkdir -p episodes
119	cat triage/

People who liked this also viewed

6 picks

Type

agent-activity-audit

45stars8forks

View on GitHub

How to install agent-activity-audit?

hyperb1iss/sibyl/agent-activity-audit

$npx -y skills add hyperb1iss/sibyl --skill agent-activity-audit

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Agent Activity Audit
2
3	This skill executes a structured pass over recent agent transcripts to learn what's working and
4	what's hurting. The original audit (May 2026) examined ~30 days of Claude Code and Codex sessions to
5	improve Sibyl itself — see `EXAMPLES.md` for the full reproducible run.
6
7	The output is a synthesis report grounded in real session evidence, plus per-group findings files
8	you can act on directly.
9
10	---
11
12	## When to use
13
14	- You maintain a system that agents call (CLI, MCP server, library, skill) and want signal beyond
15	"did it work?"
16	- You suspect agents are stumbling on something but can't name what.
17	- A planning cycle is about to start and you want product priorities grounded in usage data, not
18	vibes.
19	- A new release shipped and you want to see how it landed in the wild.
20
21	Not for: general code review, security audits, performance benchmarking. This skill reads
22	session transcripts; it doesn't analyze code.
23
24	---
25
26	## Agent rules (READ FIRST)
27
28	1. Always write artifacts under `contexts/<analysis-name>-<date>/`. Keep raw scans, episode
29	extracts, and findings in one tree so the analysis is reproducible and the user can replay or
30	extend it.
31
32	2. Filter early, filter hard. Most transcripts are noise. Triage with cheap grep before spinning
33	up parallel subagents — the goal is to give each subagent ~50-100 KB of focused episode data, not
34	raw multi-MB JSONLs.
35
36	3. Partition by date for the swarm. Date-based partitions are mutually exclusive, cover the full
37	window, and make convergence across groups easy to spot (same theme in 4+ date ranges = durable
38	issue).
39
40	4. Each subagent writes findings to a file. Don't let agents return giant prose back to the main
41	thread. Their job: produce `findings/group_<X>.md`, return a ≤250-word summary.
42
43	5. Convergence-first synthesis. A pain point in 4+ groups is durable. Single-group findings
44	warrant a sanity check before they're elevated. Count evidence; don't trust impressions.
45
46	6. Verify before recommending fixes. Inspect current source for the surfaces the audit
47	implicates. A finding like "the CLI rejects `--kind gotcha`" should point at the enum's actual
48	location.
49
50	7. Capture durable learnings to Sibyl after synthesis. The point is to feed back into the
51	product graph; use `sibyl remember --kind pattern` (or `--kind decision`) on the substantive
52	findings.
53
54	---
55
56	## The Workflow
57
58	```
59	inventory → triage → extract episodes → parallel swarm → synthesis → capture
60	```
61
62	Each step has fall-back behavior if data shape varies between Claude and Codex transcripts.
63
64	### Step 1: Inventory
65
66	Find all JSONLs in the target window. Claude lives in
67	`~/.claude/projects/<project-slug>/<uuid>.jsonl`, Codex lives in
68	`~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl`. Filter by mtime.
69
70	```bash
71	mkdir -p contexts/<name>-$(date +%F)/triage
72	cd contexts/<name>-$(date +%F)
73
74	# Last 30 days
75	CUTOFF=$(date -v-30d +%F 2>/dev/null \|\| date -d '30 days ago' +%F) # BSD (macOS) or GNU date
76	find ~/.claude/projects -name '*.jsonl' -newermt "$CUTOFF" > triage/claude_files.txt
77	find ~/.codex/sessions -name '*.jsonl' -newermt "$CUTOFF" > triage/codex_files.txt
78	cat triage/claude_files.txt triage/codex_files.txt > triage/all_files.txt
79
80	wc -l triage/*.txt
81	```
82
83	### Step 2: Triage with the scanner
84
85	Use `scripts/scan.py` (ships with this skill) to extract per-file statistics: total events, target
86	tool-call counts, error counts, user corrections. Runs in parallel.
87
88	```bash
89	SKILL_DIR=$(dirname "$(realpath "$0")") # or hard-code the path
90	cat triage/all_files.txt \| xargs -P 12 -n 5 python3 "$SKILL_DIR/scripts/scan.py" \
91	--target <tool-or-skill-keyword> > triage/scan_results.jsonl
92	```
93
94	The scanner detects three shapes of usage:
95
96	- CLI: Bash/exec_command calls whose command line matches the target's CLI name
97	- MCP: tool names matching `mcp__<target>__` or `<target>_`
98	- Skill: Skill invocations whose name matches the target
99
100	Look at the output to confirm signal quality before proceeding.
101
102	### Step 3: Extract focused episodes
103
104	Each "episode" is a target-tool call plus its preceding user message, assistant text, and tool
105	result. Use `scripts/extract_episodes.py` to write per-session markdown files (~10-30 KB each, vs
106	the 50-500 KB raw transcripts).
107
108	```bash
109	# Filter to files with actual usage
110	python3 -c "
111	import json
112	for line in open('triage/scan_results.jsonl'):
113	r = json.loads(line)
114	if r.get('target_total', 0) > 0:
115	print(r['path'])
116	" > triage/using_files.txt
117
118	mkdir -p episodes
119	cat triage/

People who liked this also viewed

6 picks

Type