clj-skill-eval

Name: clj-skill-eval
Author: humorless

33installs0forks

How to install clj-skill-eval?

humorless/clj-native-agent/clj-skill-eval

$npx -y skills add humorless/clj-native-agent --skill clj-skill-eval

Installs into the current project.

Summary

Evaluate Clojure skills through rigorous real-project testing with and without skills enabled. Use this skill whenever you need to assess whether a Clojure skill (like clj-discover, clj-debug, clj-replace) actually improves development outcomes. Compare behavior differences via c

Files · 1

View on GitHub

SKILL.md

1	# Clojure Skill Evaluation Framework
2
3	Evaluating Clojure skills requires real projects with measurable behavior differences, not hypothetical assumptions. This skill enforces a rigorous evaluation methodology that compares with-skill vs without-skill development on identical tasks.
4
5	## Core Principle
6
7	Skills only matter if they change actual behavior.
8
9	The evaluation method:
10	1. Create two isolated projects (with/without skills)
11	2. Execute identical real development tasks in both
12	3. Compare: code quality, development time, documentation depth, test comprehensiveness
13	4. Analyze context-logs to observe behavioral differences (REPL usage, exploration patterns, debugging approach)
14
15	---
16
17	## Phase 1: Evaluation Planning
18
19	### Define the Skill(s) to Evaluate
20
21	Input:
22	- Skill name and path (e.g., `clj-discover`, `clj-debug`)
23	- What behavior should the skill change? (e.g., "Should shift from log-based debugging to REPL-based debugging")
24
25	Output:
26	- Clear evaluation hypothesis (not vague assumptions)
27	- List of observable behavioral differences to measure
28
29	### Design a Realistic Development Task
30
31	Requirements:
32	- Real project scenario - Not toy examples or contrived problems
33	- Concrete requirements - Clear success criteria, not ambiguous specs
34	- Medium complexity - Enough to showcase skill value, not trivial (5-15 min per version)
35	- Type diversity - Different task types reveal different skill strengths:
36	- Discovery task: "Implement feature using unfamiliar library/API" (tests clj-discover)
37	- Debugging task: "Fix a bug in existing system" (tests clj-debug)
38	- Exploration task: "Understand and modify macro behavior" (tests clj-discover)
39	- Integration task: "Use Java library via interop" (tests clj-discover)
40
41	Example Task:
42	```
43	Implement GET /system-stats API endpoint that returns CPU usage and memory info.
44	- Use OSHI library for system statistics
45	- Return JSON with cpu-usage (0.0-1.0) and memory-info {total, available}
46	- Include at least one passing test
47	```
48
49	### Create Evaluation Criteria
50
51	Quantitative:
52	- Execution time (seconds, minutes)
53	- Token usage
54	- Lines of code
55	- Number of documentation files
56	- Test coverage
57
58	Qualitative:
59	- API choice (found optimal path or basic path?)
60	- Code quality (comments, clarity, type safety)
61	- Test defensiveness (what scenarios are covered?)
62	- Documentation depth (decisions explained or just results?)
63	- Behavioral patterns (REPL usage? Logs added? Files read?)
64
65	---
66
67	## Phase 2: Project Setup
68
69	### Create Two Isolated Projects
70
71	Use `neil` to generate identical Clojure Stack Lite projects:
72
73	```bash
74	neil new io.github.abogoyavlensky/clojure-stack-lite my-test-without-skill
75	neil new io.github.abogoyavlensky/clojure-stack-lite my-test-with-skill
76	```
77
78	Then in each:
79	```bash
80	cd my-test-without-skill && neil add nrepl
81	cd my-test-with-skill && neil add nrepl
82	```
83
84	Why isolation?
85	- No shared state or configuration
86	- Can run in parallel without interference
87	- Each gets clean Integrant system with (reset)
88
89	### Create Task Description
90
91	Create `TASK.md` in both projects with identical requirements:
92
93	```markdown
94	# Task: [Implementation Task Name]
95
96	## Objective
97	[Clear, concrete objective]
98
99	## Requirements
100	[Bullet points with specific requirements]
101
102	## Success Criteria
103	✓ Criterion 1
104	✓ Criterion 2
105	```
106
107	---
108
109	## Phase 3: Parallel Execution with Subagents
110
111	### Critical Permission Setting
112
113	When running subagents in Claude Code, use `--dangerously-skip-permissions` to avoid permission constraint blocks:
114
115	```bash
116	claude --dangerously-skip-permissions [your-claude-code-command]
117	```
118
119	Why this is necessary: Skills may use tools (brepl, WebSearch, file operations) that require explicit permission. Without this flag, subagents may abort with "permission constraints" instead of completing the task.
120
121	### Launch Both Versions in Parallel
122
123	Create two subagents simultaneously (same turn) to ensure comparable timing:
124
125	Subagent 1 (WITHOUT skills):
126	```
127	Execute this task in /path/to/my-test-without-skill:
128
129	1. Read TASK.md to understand requirements
130	2. Complete the implementation task
131	3. Save all final code files to outputs/
132	4. Record your approach in outputs/APPROACH.md:
133	- How did you discover what to do?
134	- What APIs/libraries did you choose and why?
135	- What debugging/exploration did you do?
136	- How much time was spent coding vs exploring?
137	5. DO NOT use any Clojure skills
138
139	Focus: Complete the task efficiently.
140	```
141
142	Subagent 2 (WITH skills):

People who liked this also viewed

6 picks

Type

clj-skill-eval

33installs0forks

View on GitHub

How to install clj-skill-eval?

humorless/clj-native-agent/clj-skill-eval

$npx -y skills add humorless/clj-native-agent --skill clj-skill-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Clojure Skill Evaluation Framework
2
3	Evaluating Clojure skills requires real projects with measurable behavior differences, not hypothetical assumptions. This skill enforces a rigorous evaluation methodology that compares with-skill vs without-skill development on identical tasks.
4
5	## Core Principle
6
7	Skills only matter if they change actual behavior.
8
9	The evaluation method:
10	1. Create two isolated projects (with/without skills)
11	2. Execute identical real development tasks in both
12	3. Compare: code quality, development time, documentation depth, test comprehensiveness
13	4. Analyze context-logs to observe behavioral differences (REPL usage, exploration patterns, debugging approach)
14
15	---
16
17	## Phase 1: Evaluation Planning
18
19	### Define the Skill(s) to Evaluate
20
21	Input:
22	- Skill name and path (e.g., `clj-discover`, `clj-debug`)
23	- What behavior should the skill change? (e.g., "Should shift from log-based debugging to REPL-based debugging")
24
25	Output:
26	- Clear evaluation hypothesis (not vague assumptions)
27	- List of observable behavioral differences to measure
28
29	### Design a Realistic Development Task
30
31	Requirements:
32	- Real project scenario - Not toy examples or contrived problems
33	- Concrete requirements - Clear success criteria, not ambiguous specs
34	- Medium complexity - Enough to showcase skill value, not trivial (5-15 min per version)
35	- Type diversity - Different task types reveal different skill strengths:
36	- Discovery task: "Implement feature using unfamiliar library/API" (tests clj-discover)
37	- Debugging task: "Fix a bug in existing system" (tests clj-debug)
38	- Exploration task: "Understand and modify macro behavior" (tests clj-discover)
39	- Integration task: "Use Java library via interop" (tests clj-discover)
40
41	Example Task:
42	```
43	Implement GET /system-stats API endpoint that returns CPU usage and memory info.
44	- Use OSHI library for system statistics
45	- Return JSON with cpu-usage (0.0-1.0) and memory-info {total, available}
46	- Include at least one passing test
47	```
48
49	### Create Evaluation Criteria
50
51	Quantitative:
52	- Execution time (seconds, minutes)
53	- Token usage
54	- Lines of code
55	- Number of documentation files
56	- Test coverage
57
58	Qualitative:
59	- API choice (found optimal path or basic path?)
60	- Code quality (comments, clarity, type safety)
61	- Test defensiveness (what scenarios are covered?)
62	- Documentation depth (decisions explained or just results?)
63	- Behavioral patterns (REPL usage? Logs added? Files read?)
64
65	---
66
67	## Phase 2: Project Setup
68
69	### Create Two Isolated Projects
70
71	Use `neil` to generate identical Clojure Stack Lite projects:
72
73	```bash
74	neil new io.github.abogoyavlensky/clojure-stack-lite my-test-without-skill
75	neil new io.github.abogoyavlensky/clojure-stack-lite my-test-with-skill
76	```
77
78	Then in each:
79	```bash
80	cd my-test-without-skill && neil add nrepl
81	cd my-test-with-skill && neil add nrepl
82	```
83
84	Why isolation?
85	- No shared state or configuration
86	- Can run in parallel without interference
87	- Each gets clean Integrant system with (reset)
88
89	### Create Task Description
90
91	Create `TASK.md` in both projects with identical requirements:
92
93	```markdown
94	# Task: [Implementation Task Name]
95
96	## Objective
97	[Clear, concrete objective]
98
99	## Requirements
100	[Bullet points with specific requirements]
101
102	## Success Criteria
103	✓ Criterion 1
104	✓ Criterion 2
105	```
106
107	---
108
109	## Phase 3: Parallel Execution with Subagents
110
111	### Critical Permission Setting
112
113	When running subagents in Claude Code, use `--dangerously-skip-permissions` to avoid permission constraint blocks:
114
115	```bash
116	claude --dangerously-skip-permissions [your-claude-code-command]
117	```
118
119	Why this is necessary: Skills may use tools (brepl, WebSearch, file operations) that require explicit permission. Without this flag, subagents may abort with "permission constraints" instead of completing the task.
120
121	### Launch Both Versions in Parallel
122
123	Create two subagents simultaneously (same turn) to ensure comparable timing:
124
125	Subagent 1 (WITHOUT skills):
126	```
127	Execute this task in /path/to/my-test-without-skill:
128
129	1. Read TASK.md to understand requirements
130	2. Complete the implementation task
131	3. Save all final code files to outputs/
132	4. Record your approach in outputs/APPROACH.md:
133	- How did you discover what to do?
134	- What APIs/libraries did you choose and why?
135	- What debugging/exploration did you do?
136	- How much time was spent coding vs exploring?
137	5. DO NOT use any Clojure skills
138
139	Focus: Complete the task efficiently.
140	```
141
142	Subagent 2 (WITH skills):

People who liked this also viewed

6 picks

Type