analyze-evals

Name: analyze-evals
Author: microsoft

372stars76forks

How to install analyze-evals?

microsoft/skills-for-copilot-studio/analyze-evals

$npx -y skills add microsoft/skills-for-copilot-studio --skill analyze-evals

Installs into the current project.

Summary

Analyze exported evaluation results from Copilot Studio's Evaluate tab. The user provides a CSV file exported from the Copilot Studio UI; this skill parses it, identifies failures, and proposes YAML fixes. No API access or published agent required — just the exported CSV.

Files · 1

View on GitHub

SKILL.md

1	# Analyze Copilot Studio Evaluation Results
2
3	Analyze evaluation results exported from the Copilot Studio UI as CSV.
4
5	## Phase 1: Get Results
6
7	1. Ask the user for the CSV file path if not already provided. The file is typically exported from Copilot Studio's Evaluate tab and named `Evaluate <agent name> <date>.csv` in their Downloads folder.
8
9	2. Read the CSV file. The in-product evaluation CSV has these columns:
10
11	\| Column \| Meaning \|
12	\|--------\|---------\|
13	\| `question` \| The test utterance \|
14	\| `expectedResponse` \| Expected response (may be empty) \|
15	\| `actualResponse` \| What the agent responded \|
16	\| `testMethodType_1` \| Eval method (e.g., `GeneralQuality`) \|
17	\| `result_1` \| `Pass` or `Fail` \|
18	\| `passingScore_1` \| Score threshold (may be empty) \|
19	\| `explanation_1` \| Why it passed/failed (e.g., "Seems relevant; Seems incomplete; Knowledge sources not cited") \|
20
21	The `_1` suffix indicates the first eval method. There may be additional methods (`_2`, `_3`, etc.) with the same column pattern.
22
23	## Phase 2: Analyze Results
24
25	1. Focus on failed evaluations (`result_1` = `Fail`, or any `result_N` = `Fail`).
26
27	2. For each failure, use the `explanation` column to understand the issue:
28	- "Question not answered" — The agent couldn't handle the question. Check if there's a matching topic or knowledge source.
29	- "Knowledge sources not cited" — The agent responded but didn't cite sources. Check knowledge source configuration and `SearchAndSummarizeContent` nodes.
30	- "Seems incomplete" — The response was partial. Check topic flow for early exits, missing branches, or incomplete `SendActivity` messages.
31	- Error messages in `actualResponse` (e.g., `GenAIToolPlannerRateLimitReached`) — These are runtime errors, not authoring issues. Flag them to the user as transient failures to retry.
32
33	## Phase 3: Propose Fixes
34
35	1. For each failure, identify the relevant YAML file(s):
36	- Auto-discover the agent: `Glob: **/agent.mcs.yml`
37	- Find the relevant topic by matching the test utterance against trigger phrases and model descriptions
38	- Read the topic file to understand the current flow
39
40	2. Propose specific YAML changes to fix each failure. Present them to the user as a summary:
41	- Which test(s) failed and why
42	- Which file(s) need changes
43	- What the proposed change is (show the diff)
44
45	3. Wait for user decision. The user can:
46	- Accept all — apply all proposed changes
47	- Accept partially — apply only some changes (ask which ones)
48	- Reject — discard proposed changes and discuss alternative approaches
49
50	4. Apply accepted changes using the Edit tool. After applying, remind the user to push and publish again before re-running evaluations.

People who liked this also viewed

6 picks

Type

analyze-evals

372stars76forks

View on GitHub

How to install analyze-evals?

microsoft/skills-for-copilot-studio/analyze-evals

$npx -y skills add microsoft/skills-for-copilot-studio --skill analyze-evals

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Analyze Copilot Studio Evaluation Results
2
3	Analyze evaluation results exported from the Copilot Studio UI as CSV.
4
5	## Phase 1: Get Results
6
7	1. Ask the user for the CSV file path if not already provided. The file is typically exported from Copilot Studio's Evaluate tab and named `Evaluate <agent name> <date>.csv` in their Downloads folder.
8
9	2. Read the CSV file. The in-product evaluation CSV has these columns:
10
11	\| Column \| Meaning \|
12	\|--------\|---------\|
13	\| `question` \| The test utterance \|
14	\| `expectedResponse` \| Expected response (may be empty) \|
15	\| `actualResponse` \| What the agent responded \|
16	\| `testMethodType_1` \| Eval method (e.g., `GeneralQuality`) \|
17	\| `result_1` \| `Pass` or `Fail` \|
18	\| `passingScore_1` \| Score threshold (may be empty) \|
19	\| `explanation_1` \| Why it passed/failed (e.g., "Seems relevant; Seems incomplete; Knowledge sources not cited") \|
20
21	The `_1` suffix indicates the first eval method. There may be additional methods (`_2`, `_3`, etc.) with the same column pattern.
22
23	## Phase 2: Analyze Results
24
25	1. Focus on failed evaluations (`result_1` = `Fail`, or any `result_N` = `Fail`).
26
27	2. For each failure, use the `explanation` column to understand the issue:
28	- "Question not answered" — The agent couldn't handle the question. Check if there's a matching topic or knowledge source.
29	- "Knowledge sources not cited" — The agent responded but didn't cite sources. Check knowledge source configuration and `SearchAndSummarizeContent` nodes.
30	- "Seems incomplete" — The response was partial. Check topic flow for early exits, missing branches, or incomplete `SendActivity` messages.
31	- Error messages in `actualResponse` (e.g., `GenAIToolPlannerRateLimitReached`) — These are runtime errors, not authoring issues. Flag them to the user as transient failures to retry.
32
33	## Phase 3: Propose Fixes
34
35	1. For each failure, identify the relevant YAML file(s):
36	- Auto-discover the agent: `Glob: **/agent.mcs.yml`
37	- Find the relevant topic by matching the test utterance against trigger phrases and model descriptions
38	- Read the topic file to understand the current flow
39
40	2. Propose specific YAML changes to fix each failure. Present them to the user as a summary:
41	- Which test(s) failed and why
42	- Which file(s) need changes
43	- What the proposed change is (show the diff)
44
45	3. Wait for user decision. The user can:
46	- Accept all — apply all proposed changes
47	- Accept partially — apply only some changes (ask which ones)
48	- Reject — discard proposed changes and discuss alternative approaches
49
50	4. Apply accepted changes using the Edit tool. After applying, remind the user to push and publish again before re-running evaluations.

People who liked this also viewed

6 picks

Type