create-eval

Name: create-eval
Author: microsoft

372stars76forks

How to install create-eval?

microsoft/skills-for-copilot-studio/create-eval

$npx -y skills add microsoft/skills-for-copilot-studio --skill create-eval

Installs into the current project.

Summary

Create plugin development eval scenarios (JSON files with natural prompts and deterministic checks for testing plugin skills). NOT for Copilot Studio in-product evaluation — use /copilot-studio:create-eval-set for that.

Files · 1

View on GitHub

SKILL.md

1	# Create Scenario Eval
2
3	Guide the user through creating eval test cases for a Copilot Studio plugin scenario. Evals test end-to-end scenarios with natural prompts — the request routes through sub-agents (e.g., Author agent) which invoke skills internally.
4
5	## How the eval system works
6
7	The eval harness (`evals/evaluate.py`) works by:
8	1. Copying a fixture agent into a temp workspace
9	2. Running `claude -p "<prompt>"` with a PreToolUse hook that traces skill invocations inside sub-agents
10	3. Checking routing (which agents and skills were invoked), output files, and response text against deterministic checks
11	4. Producing a JSON results file and HTML report
12
13	## What can be tested right now
14
15	Authoring scenarios that produce YAML files (topics, agents, knowledge sources, etc.) are the best candidates. The harness supports these check types:
16
17	\| Check \| What it validates \| Use for \|
18	\|-------\|------------------\|---------\|
19	\| `agent_invoked` \| Expected sub-agent was dispatched (e.g., Author agent) \| Routing verification \|
20	\| `agent_not_invoked` \| Unwanted sub-agents were NOT dispatched \| Routing verification \|
21	\| `skill_invoked` \| Expected skill was invoked (traced inside sub-agents via hook) \| Skill routing \|
22	\| `skill_not_invoked` \| Unwanted skills were NOT invoked \| Skill routing \|
23	\| `files_created` \| Expected files were created/modified (glob pattern) \| All authoring scenarios \|
24	\| `schema_validate` \| Full Copilot Studio schema validation (kind, required fields, IDs, Power Fx, scopes) \| All YAML-producing scenarios \|
25	\| `yaml_structure` \| Specific YAML path has expected value, min array length, or contains string \| Structural assertions \|
26	\| `content_contains` \| Keywords from prompt appear in output files \| Domain relevance \|
27	\| `no_placeholders` \| No `_REPLACE`, `TODO`, or `FIXME` markers left \| Template completion \|
28	\| `stdout_contains` \| CLI response text contains expected strings \| Reference/info scenarios \|
29	\| `stdout_not_contains` \| CLI response does NOT contain error strings \| Error absence \|
30	\| `exit_code` \| CLI exited with expected code \| All scenarios \|
31	\| `yaml_unchanged` \| Specific file or YAML path was NOT modified \| Preservation testing \|
32
33	Note: `no_placeholders` runs automatically when any `.mcs.yml` file is changed, unless explicitly set to `false`.
34
35	Not yet testable: Integration scenarios that call external APIs (chat-directline, manage-agent) — these need script mocking which isn't implemented yet.
36
37	## Available fixtures
38
39	Fixtures are pre-built agent directories in `evals/fixtures/`:
40
41	- basic-agent — Minimal agent with `GenerativeActionsEnabled: false`, one Greeting topic. Use for most authoring evals.
42	- agent-with-mcp-action — Same as basic-agent plus two MCP action files. Use for action-editing evals.
43	- empty-workspace — No agent files. Use for negative-path testing.
44
45	If the scenario needs a richer agent (e.g., existing topics to modify, knowledge sources, actions), note that the fixture would need to be created first.
46
47	## Instructions
48
49	1. Identify the target scenario. If `$ARGUMENTS` is provided, use it as the scenario name. Otherwise ask the user what scenario they want to test (e.g., "topic creation", "agent settings", "knowledge sources").
50
51	2. Read relevant skill SKILL.md files to understand what the scenario covers:
52	```
53	Glob: skills/*/SKILL.md
54	```
55	Understand: What skills are involved? What YAML kinds? What files get created/modified?
56
57	3. Check if evals already exist:
58	```
59	Glob: evals/scenarios/<scenario-name>.json
60	```
61	If yes, read them and offer to add more test cases. Note the highest existing eval ID.
62
63	4. Guide the user through creating test cases. For each eval, gather:
64
65	- name: Short descriptive title (e.g., "IT support topic with OnRecognizedIntent trigger")
66	- prompt: A natural language prompt — what a real user would say. Do NOT prefix with "Use the X skill to...".
67	- fixture: Which fixture agent to use (default: `basic-agent`)
68	- checks: What to validate about the routing and output
69
70	5. Help the user define checks. Based on the scenario type:
71
72	For topic-creation scenarios:
73	```json
74	{
75	"agent_invoked": "copilot-studio:Copilot Studio Author",
76	"skill_invoked": "copilot-studio:new-topic",
77	"files_created": [{"pattern": "topics/*.topic.mcs.yml", "min_count": 1}],
78	"schema_validate": true,
79	"yaml_structure": [
80	{"path": "kind", "equals": "AdaptiveDialog"},
81	{"path": "beginDialog.kind", "equals": "<trigger-type>"}
82	],
83	"content_contains": ["<domain keywords>"],
84	"no_placeholders": true
85	}
86	```
87
88	For **a

People who liked this also viewed

6 picks

Type

create-eval

372stars76forks

View on GitHub

How to install create-eval?

microsoft/skills-for-copilot-studio/create-eval

$npx -y skills add microsoft/skills-for-copilot-studio --skill create-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Create Scenario Eval
2
3	Guide the user through creating eval test cases for a Copilot Studio plugin scenario. Evals test end-to-end scenarios with natural prompts — the request routes through sub-agents (e.g., Author agent) which invoke skills internally.
4
5	## How the eval system works
6
7	The eval harness (`evals/evaluate.py`) works by:
8	1. Copying a fixture agent into a temp workspace
9	2. Running `claude -p "<prompt>"` with a PreToolUse hook that traces skill invocations inside sub-agents
10	3. Checking routing (which agents and skills were invoked), output files, and response text against deterministic checks
11	4. Producing a JSON results file and HTML report
12
13	## What can be tested right now
14
15	Authoring scenarios that produce YAML files (topics, agents, knowledge sources, etc.) are the best candidates. The harness supports these check types:
16
17	\| Check \| What it validates \| Use for \|
18	\|-------\|------------------\|---------\|
19	\| `agent_invoked` \| Expected sub-agent was dispatched (e.g., Author agent) \| Routing verification \|
20	\| `agent_not_invoked` \| Unwanted sub-agents were NOT dispatched \| Routing verification \|
21	\| `skill_invoked` \| Expected skill was invoked (traced inside sub-agents via hook) \| Skill routing \|
22	\| `skill_not_invoked` \| Unwanted skills were NOT invoked \| Skill routing \|
23	\| `files_created` \| Expected files were created/modified (glob pattern) \| All authoring scenarios \|
24	\| `schema_validate` \| Full Copilot Studio schema validation (kind, required fields, IDs, Power Fx, scopes) \| All YAML-producing scenarios \|
25	\| `yaml_structure` \| Specific YAML path has expected value, min array length, or contains string \| Structural assertions \|
26	\| `content_contains` \| Keywords from prompt appear in output files \| Domain relevance \|
27	\| `no_placeholders` \| No `_REPLACE`, `TODO`, or `FIXME` markers left \| Template completion \|
28	\| `stdout_contains` \| CLI response text contains expected strings \| Reference/info scenarios \|
29	\| `stdout_not_contains` \| CLI response does NOT contain error strings \| Error absence \|
30	\| `exit_code` \| CLI exited with expected code \| All scenarios \|
31	\| `yaml_unchanged` \| Specific file or YAML path was NOT modified \| Preservation testing \|
32
33	Note: `no_placeholders` runs automatically when any `.mcs.yml` file is changed, unless explicitly set to `false`.
34
35	Not yet testable: Integration scenarios that call external APIs (chat-directline, manage-agent) — these need script mocking which isn't implemented yet.
36
37	## Available fixtures
38
39	Fixtures are pre-built agent directories in `evals/fixtures/`:
40
41	- basic-agent — Minimal agent with `GenerativeActionsEnabled: false`, one Greeting topic. Use for most authoring evals.
42	- agent-with-mcp-action — Same as basic-agent plus two MCP action files. Use for action-editing evals.
43	- empty-workspace — No agent files. Use for negative-path testing.
44
45	If the scenario needs a richer agent (e.g., existing topics to modify, knowledge sources, actions), note that the fixture would need to be created first.
46
47	## Instructions
48
49	1. Identify the target scenario. If `$ARGUMENTS` is provided, use it as the scenario name. Otherwise ask the user what scenario they want to test (e.g., "topic creation", "agent settings", "knowledge sources").
50
51	2. Read relevant skill SKILL.md files to understand what the scenario covers:
52	```
53	Glob: skills/*/SKILL.md
54	```
55	Understand: What skills are involved? What YAML kinds? What files get created/modified?
56
57	3. Check if evals already exist:
58	```
59	Glob: evals/scenarios/<scenario-name>.json
60	```
61	If yes, read them and offer to add more test cases. Note the highest existing eval ID.
62
63	4. Guide the user through creating test cases. For each eval, gather:
64
65	- name: Short descriptive title (e.g., "IT support topic with OnRecognizedIntent trigger")
66	- prompt: A natural language prompt — what a real user would say. Do NOT prefix with "Use the X skill to...".
67	- fixture: Which fixture agent to use (default: `basic-agent`)
68	- checks: What to validate about the routing and output
69
70	5. Help the user define checks. Based on the scenario type:
71
72	For topic-creation scenarios:
73	```json
74	{
75	"agent_invoked": "copilot-studio:Copilot Studio Author",
76	"skill_invoked": "copilot-studio:new-topic",
77	"files_created": [{"pattern": "topics/*.topic.mcs.yml", "min_count": 1}],
78	"schema_validate": true,
79	"yaml_structure": [
80	{"path": "kind", "equals": "AdaptiveDialog"},
81	{"path": "beginDialog.kind", "equals": "<trigger-type>"}
82	],
83	"content_contains": ["<domain keywords>"],
84	"no_placeholders": true
85	}
86	```
87
88	For **a

People who liked this also viewed

6 picks

Type