run-eval

Name: run-eval
Author: microsoft

372stars76forks

How to install run-eval?

microsoft/skills-for-copilot-studio/run-eval

$npx -y skills add microsoft/skills-for-copilot-studio --skill run-eval

Installs into the current project.

Summary

Run evaluations against a Copilot Studio agent via the Power Platform Evaluation API. Works on DRAFT agents — no publish step required. Lists test sets, starts a run, polls until complete, fetches results, and proposes YAML fixes for failures. Use when the user wants to test agen

Files · 1

View on GitHub

SKILL.md

1	# Run Evaluation (PPAPI)
2
3	Run evaluations against a Copilot Studio agent's draft — no publish needed.
4
5	The caller (test agent) must provide `--client-id` and `--workspace`. If you don't have the client ID, return immediately and tell the caller to run `test-auth` first.
6
7	All eval-api commands run in the foreground. NEVER use `run_in_background`.
8
9	## Step 1: List test sets and let the user choose
10
11	```bash
12	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js list-testsets --workspace <path> --client-id <id>
13	```
14
15	- No test sets found: Tell the user to create one in Copilot Studio (Evaluate tab > New evaluation). Stop.
16	- One test set: Tell the user which one you're using and proceed.
17	- Multiple test sets: Show them all and ask the user to pick. Do not proceed until they answer.
18
19	## Step 2: Ask about authenticated execution — MANDATORY, do not skip
20
21	You MUST ask this question and wait for the user's answer before starting the run.
22
23	Ask the user:
24
25	> Does your agent use authenticated knowledge sources or connector actions (tools) that require user identity?
26	> If so, you'll need to provide a connection ID — without it, the eval runs anonymously and tools and knowledge sources will not be used.
27	>
28	> How to obtain the connection ID:
29	> 1. Go to https://make.powerautomate.com
30	> 2. Open Connections from the side menu
31	> 3. Select the relevant Microsoft Copilot Studio connection
32	> 4. Copy the connection ID from the URL (the GUID segment after `/connections/`)
33	>
34	> If your agent doesn't use authenticated knowledge or tools, you can skip this.
35
36	Do not proceed to Step 3 until the user responds.
37
38	## Step 3: Start the run
39
40	```bash
41	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js start-run --workspace <path> --client-id <id> --testset-id <id> --run-name "Draft eval <date>"
42	```
43
44	Add `--connection-id <id>` if the user provided a connection ID in Step 2.
45
46	Add `--published` only if the user explicitly asked for published-bot testing.
47
48	## Step 4: Poll until complete
49
50	```bash
51	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-run --workspace <path> --client-id <id> --run-id <runId>
52	```
53
54	Poll every 15-30 seconds. Report progress: "Processing: 3/10 test cases..."
55
56	Stop when `state` is `Completed`, `Failed`, `Abandoned`, or `Cancelled`.
57
58	## Step 5: Fetch and analyze results
59
60	```bash
61	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-results --workspace <path> --client-id <id> --run-id <runId>
62	```
63
64	Present a summary table (total, passed, failed, errors). For failures:
65
66	\| Metric \| What to check \|
67	\|--------\|---------------\|
68	\| `GeneralQuality` Fail \| Which of relevance/completeness/groundedness/abstention failed \|
69	\| `ExactMatch` Fail \| Score 0.0–1.0 \|
70	\| `CapabilityUse` Fail \| `missingInvocationSteps` \|
71	\| `Error` status \| `errorReason` — often a test set config issue, not a YAML issue \|
72
73	## Step 6: Propose fixes (if failures found)
74
75	For YAML authoring failures: find the relevant topic, read it, propose specific edits. Wait for user approval before applying.
76
77	After applying: offer to push and re-run (go back to Step 3).

People who liked this also viewed

6 picks

Type

run-eval

372stars76forks

View on GitHub

How to install run-eval?

microsoft/skills-for-copilot-studio/run-eval

$npx -y skills add microsoft/skills-for-copilot-studio --skill run-eval

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Run Evaluation (PPAPI)
2
3	Run evaluations against a Copilot Studio agent's draft — no publish needed.
4
5	The caller (test agent) must provide `--client-id` and `--workspace`. If you don't have the client ID, return immediately and tell the caller to run `test-auth` first.
6
7	All eval-api commands run in the foreground. NEVER use `run_in_background`.
8
9	## Step 1: List test sets and let the user choose
10
11	```bash
12	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js list-testsets --workspace <path> --client-id <id>
13	```
14
15	- No test sets found: Tell the user to create one in Copilot Studio (Evaluate tab > New evaluation). Stop.
16	- One test set: Tell the user which one you're using and proceed.
17	- Multiple test sets: Show them all and ask the user to pick. Do not proceed until they answer.
18
19	## Step 2: Ask about authenticated execution — MANDATORY, do not skip
20
21	You MUST ask this question and wait for the user's answer before starting the run.
22
23	Ask the user:
24
25	> Does your agent use authenticated knowledge sources or connector actions (tools) that require user identity?
26	> If so, you'll need to provide a connection ID — without it, the eval runs anonymously and tools and knowledge sources will not be used.
27	>
28	> How to obtain the connection ID:
29	> 1. Go to https://make.powerautomate.com
30	> 2. Open Connections from the side menu
31	> 3. Select the relevant Microsoft Copilot Studio connection
32	> 4. Copy the connection ID from the URL (the GUID segment after `/connections/`)
33	>
34	> If your agent doesn't use authenticated knowledge or tools, you can skip this.
35
36	Do not proceed to Step 3 until the user responds.
37
38	## Step 3: Start the run
39
40	```bash
41	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js start-run --workspace <path> --client-id <id> --testset-id <id> --run-name "Draft eval <date>"
42	```
43
44	Add `--connection-id <id>` if the user provided a connection ID in Step 2.
45
46	Add `--published` only if the user explicitly asked for published-bot testing.
47
48	## Step 4: Poll until complete
49
50	```bash
51	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-run --workspace <path> --client-id <id> --run-id <runId>
52	```
53
54	Poll every 15-30 seconds. Report progress: "Processing: 3/10 test cases..."
55
56	Stop when `state` is `Completed`, `Failed`, `Abandoned`, or `Cancelled`.
57
58	## Step 5: Fetch and analyze results
59
60	```bash
61	node ${CLAUDE_SKILL_DIR}/../../scripts/eval-api.bundle.js get-results --workspace <path> --client-id <id> --run-id <runId>
62	```
63
64	Present a summary table (total, passed, failed, errors). For failures:
65
66	\| Metric \| What to check \|
67	\|--------\|---------------\|
68	\| `GeneralQuality` Fail \| Which of relevance/completeness/groundedness/abstention failed \|
69	\| `ExactMatch` Fail \| Score 0.0–1.0 \|
70	\| `CapabilityUse` Fail \| `missingInvocationSteps` \|
71	\| `Error` status \| `errorReason` — often a test set config issue, not a YAML issue \|
72
73	## Step 6: Propose fixes (if failures found)
74
75	For YAML authoring failures: find the relevant topic, read it, propose specific edits. Wait for user approval before applying.
76
77	After applying: offer to push and re-run (go back to Step 3).

People who liked this also viewed

6 picks

Type