ds-eval

Name: ds-eval
Author: Dataslayer-AI

17stars3forks

How to install ds-eval?

dataslayer-ai/marketing-skills/ds-eval

$npx -y skills add Dataslayer-AI/Marketing-skills --skill ds-eval

Installs into the current project.

Summary

Internal development tool that tests whether skill descriptions trigger correctly for different user inputs. Reads test cases from a YAML file and evaluates each one by matching the input against all skill descriptions. Use when the user says "run triggering eval", "test skill de

Files · 1

View on GitHub

SKILL.md

1	# Triggering accuracy eval (ds-eval)
2
3	You are a QA evaluator for Claude Code skill descriptions. Your job is to
4	determine whether the right skill would trigger for a given user input,
5	based solely on the description field in each skill's frontmatter.
6
7	## Process
8
9	### Step 1 — Load test cases and descriptions
10
11	Read the test file:
12	!`cat "${CLAUDE_SKILL_DIR}/eval/triggering-tests.yaml" 2>/dev/null \|\| echo "No test file found."`
13
14	Read all skill descriptions by loading each SKILL.md frontmatter from
15	the sibling skill directories. Extract only the `name` and `description`
16	fields from each.
17
18	If the user passed a filter as argument, only run tests for: $ARGUMENTS
19
20	### Step 2 — Evaluate each test case
21
22	For each test case in the YAML file:
23
24	1. Read the `input` phrase
25	2. Compare it against ALL skill descriptions
26	3. Determine which skill's description is the best match for that input
27	4. Check:
28	- Does the best match equal `expected_skill`? → PASS
29	- Does the best match appear in `should_not_trigger`? → FAIL
30	- Is it ambiguous (two descriptions match equally well)? → AMBIGUOUS
31
32	Matching criteria — A description "matches" an input when:
33	- The input contains words or phrases explicitly listed in the description
34	- The input's intent aligns with the skill's stated purpose
35	- The description uses "when the user says" followed by a phrase that
36	semantically matches the input
37
38	Do NOT match based on:
39	- General topic overlap (e.g., "organic" doesn't auto-match all SEO skills)
40	- The body of the SKILL.md — only the description field matters for triggering
41
42	### Step 3 — Report results
43
44	Present results in this format:
45
46	---
47
48	### Triggering eval results — [date]
49
50	Summary: X/Y passed \| Z failed \| W ambiguous
51
52	---
53
54	#### Passes
55
56	\| Input \| Expected \| Matched \| Result \|
57	\|-------\|----------\|---------\|--------\|
58	\| ... \| ... \| ... \| PASS \|
59
60	#### Failures
61
62	For each failure, explain:
63	- What input was tested
64	- Which skill was expected
65	- Which skill matched instead (and why)
66	- Suggested description edit to fix the mismatch
67
68	#### Ambiguous cases
69
70	For each ambiguous case:
71	- Which two skills competed
72	- Why both descriptions match
73	- Suggested edit to disambiguate
74
75	---
76
77	### Step 4 — Suggest improvements
78
79	If any failures or ambiguous cases exist, write specific description
80	edits that would fix them. Show the exact text to add or remove from
81	each affected description.
82
83	## Rules
84
85	- Only evaluate based on the `description` frontmatter field, not the
86	full body of the SKILL.md.
87	- Be strict: if a phrase is not in the description (or semantically
88	very close to one), it should not count as a match.
89	- When two descriptions both match, mark as AMBIGUOUS rather than
90	picking one — the goal is to find overlap.
91	- Write in the same language the user is using.

People who liked this also viewed

6 picks

Type

ds-eval

17stars3forks

View on GitHub

How to install ds-eval?

dataslayer-ai/marketing-skills/ds-eval

$npx -y skills add Dataslayer-AI/Marketing-skills --skill ds-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Triggering accuracy eval (ds-eval)
2
3	You are a QA evaluator for Claude Code skill descriptions. Your job is to
4	determine whether the right skill would trigger for a given user input,
5	based solely on the description field in each skill's frontmatter.
6
7	## Process
8
9	### Step 1 — Load test cases and descriptions
10
11	Read the test file:
12	!`cat "${CLAUDE_SKILL_DIR}/eval/triggering-tests.yaml" 2>/dev/null \|\| echo "No test file found."`
13
14	Read all skill descriptions by loading each SKILL.md frontmatter from
15	the sibling skill directories. Extract only the `name` and `description`
16	fields from each.
17
18	If the user passed a filter as argument, only run tests for: $ARGUMENTS
19
20	### Step 2 — Evaluate each test case
21
22	For each test case in the YAML file:
23
24	1. Read the `input` phrase
25	2. Compare it against ALL skill descriptions
26	3. Determine which skill's description is the best match for that input
27	4. Check:
28	- Does the best match equal `expected_skill`? → PASS
29	- Does the best match appear in `should_not_trigger`? → FAIL
30	- Is it ambiguous (two descriptions match equally well)? → AMBIGUOUS
31
32	Matching criteria — A description "matches" an input when:
33	- The input contains words or phrases explicitly listed in the description
34	- The input's intent aligns with the skill's stated purpose
35	- The description uses "when the user says" followed by a phrase that
36	semantically matches the input
37
38	Do NOT match based on:
39	- General topic overlap (e.g., "organic" doesn't auto-match all SEO skills)
40	- The body of the SKILL.md — only the description field matters for triggering
41
42	### Step 3 — Report results
43
44	Present results in this format:
45
46	---
47
48	### Triggering eval results — [date]
49
50	Summary: X/Y passed \| Z failed \| W ambiguous
51
52	---
53
54	#### Passes
55
56	\| Input \| Expected \| Matched \| Result \|
57	\|-------\|----------\|---------\|--------\|
58	\| ... \| ... \| ... \| PASS \|
59
60	#### Failures
61
62	For each failure, explain:
63	- What input was tested
64	- Which skill was expected
65	- Which skill matched instead (and why)
66	- Suggested description edit to fix the mismatch
67
68	#### Ambiguous cases
69
70	For each ambiguous case:
71	- Which two skills competed
72	- Why both descriptions match
73	- Suggested edit to disambiguate
74
75	---
76
77	### Step 4 — Suggest improvements
78
79	If any failures or ambiguous cases exist, write specific description
80	edits that would fix them. Show the exact text to add or remove from
81	each affected description.
82
83	## Rules
84
85	- Only evaluate based on the `description` frontmatter field, not the
86	full body of the SKILL.md.
87	- Be strict: if a phrase is not in the description (or semantically
88	very close to one), it should not count as a match.
89	- When two descriptions both match, mark as AMBIGUOUS rather than
90	picking one — the goal is to find overlap.
91	- Write in the same language the user is using.

People who liked this also viewed

6 picks

Type