create-eval-set

Name: create-eval-set
Author: microsoft

372stars76forks

How to install create-eval-set?

microsoft/skills-for-copilot-studio/create-eval-set

$npx -y skills add microsoft/skills-for-copilot-studio --skill create-eval-set

Installs into the current project.

Summary

Create a test set CSV file for import into Copilot Studio's in-product Evaluate tab. Reads the agent's topics, instructions, and knowledge sources to generate meaningful test cases with appropriate graders (General quality, Compare meaning, Exact match, etc.). Use when the user a

Files · 1

View on GitHub

SKILL.md

1	# Create Evaluation Test Set
2
3	Create a test set CSV file that can be imported into Copilot Studio's Evaluate tab for in-product agent evaluation.
4
5	## Phase 1: Understand the Agent
6
7	Read the agent's YAML files to understand what it does:
8
9	1. `Glob: **/agent.mcs.yml` — find the agent
10	2. Read `agent.mcs.yml` — get the agent's instructions, description, and capabilities
11	3. Read `settings.mcs.yml` — check orchestration mode (generative vs classic)
12	4. `Glob: */topics/.mcs.yml` — list all topics
13	5. Read key topics (especially non-system ones) — understand trigger phrases, conversation flows, expected behaviors
14	6. Check for knowledge sources, actions, and connected tools
15
16	## Phase 2: Design Test Cases
17
18	Create test cases that cover:
19
20	\| Category \| What to test \| Example \|
21	\|----------\|-------------\|---------\|
22	\| Core functionality \| Main topics and capabilities \| Questions matching trigger phrases \|
23	\| Knowledge/generative \| Knowledge source responses \| Questions the agent should answer from its knowledge \|
24	\| System topics \| Greeting, Escalation, Goodbye, Thank You, Fallback \| "Hi", "I want to speak to a person", "Goodbye" \|
25	\| Edge cases \| Out-of-scope, ambiguous, off-topic \| "Tell me a joke", "Book a flight for me" \|
26	\| Boundary testing \| Things the agent should NOT do \| Actions beyond its capabilities \|
27
28	Aim for 10–25 test cases with good coverage across categories.
29
30	## Phase 3: Write the Expected Responses
31
32	The CSV import only supports two columns: `question` and `expectedResponse`. Test methods cannot be set via CSV import — they are configured in the UI after import. The default test method (General quality) is applied to all imported test cases.
33
34	Write expected responses with this in mind:
35	- For questions where you want General quality grading: write behavioral descriptions ("The response should recommend hotels in Paris with relevant details")
36	- For questions where you'll later switch to Compare meaning or Exact match in the UI: write realistic agent replies that the grader can compare against
37	- Leave `expectedResponse` empty for questions that only need General quality (it works without expected responses)
38
39	### Available test methods (configured in UI after import)
40
41	\| Test method \| What it measures \| Requires expected response? \|
42	\|-------------\|-----------------\|---------------------------\|
43	\| General quality (default) \| AI-graded quality: relevance, completeness, groundedness, abstention \| No (but recommended as a rubric) \|
44	\| Compare meaning \| Semantic similarity — compares meaning/intent \| Yes \|
45	\| Text similarity \| Cosine similarity of text \| Yes, configurable pass threshold \|
46	\| Exact match \| Character-for-character match \| Yes \|
47	\| Keyword match \| Response contains expected keywords/phrases \| Yes (keywords added in UI) \|
48	\| Capability use \| Agent called expected tools/topics \| Configured in UI \|
49	\| Custom \| Custom grader with your own instructions and labels \| Configured in UI \|
50
51	## Phase 4: Write the CSV
52
53	Write the CSV file using the Write tool. The format must be:
54
55	```csv
56	"question","expectedResponse"
57	"User question here","Expected agent response or behavioral rubric"
58	"Question without expected response",
59	```
60
61	### Column specification
62
63	\| Column \| Required \| Description \|
64	\|--------\|----------\|-------------\|
65	\| `question` \| Yes \| The user message to send to the agent. Max 1,000 characters. \|
66	\| `expectedResponse` \| No \| The expected response or behavioral rubric. Leave empty if not needed. \|
67
68	Important: The `Testing method` column is not supported on import — it is ignored. All imported test cases get the default test method (General quality). Configure other test methods in the UI after import.
69
70	### Rules
71	- Max 100 questions per test set
72	- Max 1,000 characters per question (including spaces)
73	- File must be `.csv` format
74	- Use double quotes around all values
75	- For questions that will use General quality: write expected responses as behavioral descriptions
76	- For questions that will use Compare meaning or Exact match: write expected responses as realistic agent replies
77
78	### Expected response examples
79
80	Behavioral rubric (for General quality):
81	```csv
82	"Find me a hotel in Paris","The response should include hotel recommendations in Paris with relevant details like names, locations, or prices."
83	```
84
85	Realistic reply (for Compare meaning — set method in UI after import):
86	```csv
87	"Hi there","Hello! How can I help you today?"
88	```
89
90	**Exact expected text

People who liked this also viewed

6 picks

Type

create-eval-set

372stars76forks

View on GitHub

How to install create-eval-set?

microsoft/skills-for-copilot-studio/create-eval-set

$npx -y skills add microsoft/skills-for-copilot-studio --skill create-eval-set

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Create Evaluation Test Set
2
3	Create a test set CSV file that can be imported into Copilot Studio's Evaluate tab for in-product agent evaluation.
4
5	## Phase 1: Understand the Agent
6
7	Read the agent's YAML files to understand what it does:
8
9	1. `Glob: **/agent.mcs.yml` — find the agent
10	2. Read `agent.mcs.yml` — get the agent's instructions, description, and capabilities
11	3. Read `settings.mcs.yml` — check orchestration mode (generative vs classic)
12	4. `Glob: */topics/.mcs.yml` — list all topics
13	5. Read key topics (especially non-system ones) — understand trigger phrases, conversation flows, expected behaviors
14	6. Check for knowledge sources, actions, and connected tools
15
16	## Phase 2: Design Test Cases
17
18	Create test cases that cover:
19
20	\| Category \| What to test \| Example \|
21	\|----------\|-------------\|---------\|
22	\| Core functionality \| Main topics and capabilities \| Questions matching trigger phrases \|
23	\| Knowledge/generative \| Knowledge source responses \| Questions the agent should answer from its knowledge \|
24	\| System topics \| Greeting, Escalation, Goodbye, Thank You, Fallback \| "Hi", "I want to speak to a person", "Goodbye" \|
25	\| Edge cases \| Out-of-scope, ambiguous, off-topic \| "Tell me a joke", "Book a flight for me" \|
26	\| Boundary testing \| Things the agent should NOT do \| Actions beyond its capabilities \|
27
28	Aim for 10–25 test cases with good coverage across categories.
29
30	## Phase 3: Write the Expected Responses
31
32	The CSV import only supports two columns: `question` and `expectedResponse`. Test methods cannot be set via CSV import — they are configured in the UI after import. The default test method (General quality) is applied to all imported test cases.
33
34	Write expected responses with this in mind:
35	- For questions where you want General quality grading: write behavioral descriptions ("The response should recommend hotels in Paris with relevant details")
36	- For questions where you'll later switch to Compare meaning or Exact match in the UI: write realistic agent replies that the grader can compare against
37	- Leave `expectedResponse` empty for questions that only need General quality (it works without expected responses)
38
39	### Available test methods (configured in UI after import)
40
41	\| Test method \| What it measures \| Requires expected response? \|
42	\|-------------\|-----------------\|---------------------------\|
43	\| General quality (default) \| AI-graded quality: relevance, completeness, groundedness, abstention \| No (but recommended as a rubric) \|
44	\| Compare meaning \| Semantic similarity — compares meaning/intent \| Yes \|
45	\| Text similarity \| Cosine similarity of text \| Yes, configurable pass threshold \|
46	\| Exact match \| Character-for-character match \| Yes \|
47	\| Keyword match \| Response contains expected keywords/phrases \| Yes (keywords added in UI) \|
48	\| Capability use \| Agent called expected tools/topics \| Configured in UI \|
49	\| Custom \| Custom grader with your own instructions and labels \| Configured in UI \|
50
51	## Phase 4: Write the CSV
52
53	Write the CSV file using the Write tool. The format must be:
54
55	```csv
56	"question","expectedResponse"
57	"User question here","Expected agent response or behavioral rubric"
58	"Question without expected response",
59	```
60
61	### Column specification
62
63	\| Column \| Required \| Description \|
64	\|--------\|----------\|-------------\|
65	\| `question` \| Yes \| The user message to send to the agent. Max 1,000 characters. \|
66	\| `expectedResponse` \| No \| The expected response or behavioral rubric. Leave empty if not needed. \|
67
68	Important: The `Testing method` column is not supported on import — it is ignored. All imported test cases get the default test method (General quality). Configure other test methods in the UI after import.
69
70	### Rules
71	- Max 100 questions per test set
72	- Max 1,000 characters per question (including spaces)
73	- File must be `.csv` format
74	- Use double quotes around all values
75	- For questions that will use General quality: write expected responses as behavioral descriptions
76	- For questions that will use Compare meaning or Exact match: write expected responses as realistic agent replies
77
78	### Expected response examples
79
80	Behavioral rubric (for General quality):
81	```csv
82	"Find me a hotel in Paris","The response should include hotel recommendations in Paris with relevant details like names, locations, or prices."
83	```
84
85	Realistic reply (for Compare meaning — set method in UI after import):
86	```csv
87	"Hi there","Hello! How can I help you today?"
88	```
89
90	**Exact expected text

People who liked this also viewed

6 picks

Type