run-tests-kit

Name: run-tests-kit
Author: microsoft

372stars76forks

How to install run-tests-kit?

microsoft/skills-for-copilot-studio/run-tests-kit

$npx -y skills add microsoft/skills-for-copilot-studio --skill run-tests-kit

Installs into the current project.

Summary

Run a batch test suite via the Copilot Studio Kit (Dataverse API). Uses the Power CAT Copilot Studio Kit to execute test cases against a published agent and produces pass/fail results with latencies. Requires the Kit installed in the environment, an App Registration with Datavers

Files · 1

View on GitHub

SKILL.md

1	# Run Tests via Copilot Studio Kit
2
3	Run a batch test suite against a published Copilot Studio agent using the [Power CAT Copilot Studio Kit](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit).
4
5	## Prerequisites
6
7	The user must have:
8	1. The [Copilot Studio Kit](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit) installed in their Power Platform environment
9	2. Published their agent in the Copilot Studio UI
10	3. Created a test set in the Copilot Studio Kit
11	4. An Azure App Registration with Dataverse permissions
12
13	## Phase 1: Configure Settings
14
15	1. Read `tests/settings.json` (relative to the user's project CWD) and check for missing or placeholder values (containing `YOUR_`).
16
17	2. If the file doesn't exist, create it from the template:
18	```bash
19	cp ${CLAUDE_SKILL_DIR}/../../tests/settings-example.json ./tests/settings.json
20	```
21
22	3. If values are missing, ask the user for each missing value. Explain where to find each one:
23
24	- Environment URL (`dataverse.environmentUrl`): "What is your Dataverse environment URL? Find it in Power Platform admin center or Copilot Studio > Settings > Session Details. It looks like `https://orgXXXXXX.crm.dynamics.com`"
25	- Tenant ID (`dataverse.tenantId`): "What is your Azure tenant ID? Find it in Azure Portal > Microsoft Entra ID > Overview. It's a GUID like `c87f36f7-fc65-453c-9019-0d724f21bc42`"
26	- Client ID (`dataverse.clientId`): "What is your App Registration client ID? Find it in Azure Portal > App Registrations > your app > Application (client) ID. It's a GUID."
27	- Agent Configuration ID (`testRun.agentConfigurationId`): "What is your agent configuration ID? In Copilot Studio, go to your agent > Tests tab. The ID is a GUID found in the URL or test configuration."
28	- Test Set ID (`testRun.agentTestSetId`): "What is your test set ID? In Copilot Studio, go to your agent > Tests tab > select your test set. The ID is a GUID found in the URL."
29
30	Ask for ALL missing values at once (don't ask one at a time).
31
32	4. Write `tests/settings.json` with the collected values:
33	```json
34	{
35	"dataverse": {
36	"environmentUrl": "<value>",
37	"tenantId": "<value>",
38	"clientId": "<value>"
39	},
40	"testRun": {
41	"agentConfigurationId": "<value>",
42	"agentTestSetId": "<value>"
43	}
44	}
45	```
46
47	5. If all values are already configured and valid, proceed to Phase 2.
48
49	## Phase 2: Run Tests
50
51	1. Ensure `tests/package.json` exists in the user's project. If not, copy it:
52	```bash
53	cp ${CLAUDE_SKILL_DIR}/../../tests/package.json ./tests/package.json
54	```
55
56	2. Install dependencies if `tests/node_modules/` doesn't exist:
57	```bash
58	npm install --prefix tests
59	```
60
61	3. Run the test script in the background with a 100-minute timeout (6000000ms):
62	```bash
63	node ${CLAUDE_SKILL_DIR}/../../tests/run-tests.js --config-dir ./tests
64	```
65	Use `run_in_background: true` for this command. Save the returned task ID.
66
67	4. Wait 10 seconds, then check the background task output (non-blocking check).
68
69	5. Detect the authentication state from the output:
70
71	- If the output contains "Using cached token": Authentication succeeded automatically. Tell the user: "Authentication successful (cached credentials). Tests are running, this may take several minutes..."
72
73	- If the output contains "use a web browser to open the page": Extract the URL and device code from the message. Present this prominently to the user:
74
75	> Authentication Required
76	>
77	> Open your browser to: https://microsoft.com/devicelogin
78	> Enter the code: XXXXXXXXX (extract the actual code from the output)
79	>
80	> After signing in, the tests will continue automatically.
81
82	- If the output contains an error: Report the error to the user and stop.
83
84	- If the output is empty or incomplete: Wait another 10 seconds and check again (retry up to 3 times).
85
86	6. Wait for the background task to complete (blocking). The script polls every 20 seconds until all tests finish and downloads results as a CSV.
87
88	7. Read the final output to get the success rate and CSV filename.
89
90	8. Proceed to Phase 3.
91
92	## Phase 3: Analyze Results
93
94	1. Get the results: `Glob: tests/test-results-*.csv` — read the most recent CSV file (newest by modification time).
95
96	2. Parse the CSV columns:
97	\| Column \| Meaning \|
98	\|--------\|---------\|
99	\| Test Utterance \| The user message that was t

People who liked this also viewed

6 picks

Type

run-tests-kit

372stars76forks

View on GitHub

How to install run-tests-kit?

microsoft/skills-for-copilot-studio/run-tests-kit

$npx -y skills add microsoft/skills-for-copilot-studio --skill run-tests-kit

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Run Tests via Copilot Studio Kit
2
3	Run a batch test suite against a published Copilot Studio agent using the [Power CAT Copilot Studio Kit](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit).
4
5	## Prerequisites
6
7	The user must have:
8	1. The [Copilot Studio Kit](https://github.com/microsoft/Power-CAT-Copilot-Studio-Kit) installed in their Power Platform environment
9	2. Published their agent in the Copilot Studio UI
10	3. Created a test set in the Copilot Studio Kit
11	4. An Azure App Registration with Dataverse permissions
12
13	## Phase 1: Configure Settings
14
15	1. Read `tests/settings.json` (relative to the user's project CWD) and check for missing or placeholder values (containing `YOUR_`).
16
17	2. If the file doesn't exist, create it from the template:
18	```bash
19	cp ${CLAUDE_SKILL_DIR}/../../tests/settings-example.json ./tests/settings.json
20	```
21
22	3. If values are missing, ask the user for each missing value. Explain where to find each one:
23
24	- Environment URL (`dataverse.environmentUrl`): "What is your Dataverse environment URL? Find it in Power Platform admin center or Copilot Studio > Settings > Session Details. It looks like `https://orgXXXXXX.crm.dynamics.com`"
25	- Tenant ID (`dataverse.tenantId`): "What is your Azure tenant ID? Find it in Azure Portal > Microsoft Entra ID > Overview. It's a GUID like `c87f36f7-fc65-453c-9019-0d724f21bc42`"
26	- Client ID (`dataverse.clientId`): "What is your App Registration client ID? Find it in Azure Portal > App Registrations > your app > Application (client) ID. It's a GUID."
27	- Agent Configuration ID (`testRun.agentConfigurationId`): "What is your agent configuration ID? In Copilot Studio, go to your agent > Tests tab. The ID is a GUID found in the URL or test configuration."
28	- Test Set ID (`testRun.agentTestSetId`): "What is your test set ID? In Copilot Studio, go to your agent > Tests tab > select your test set. The ID is a GUID found in the URL."
29
30	Ask for ALL missing values at once (don't ask one at a time).
31
32	4. Write `tests/settings.json` with the collected values:
33	```json
34	{
35	"dataverse": {
36	"environmentUrl": "<value>",
37	"tenantId": "<value>",
38	"clientId": "<value>"
39	},
40	"testRun": {
41	"agentConfigurationId": "<value>",
42	"agentTestSetId": "<value>"
43	}
44	}
45	```
46
47	5. If all values are already configured and valid, proceed to Phase 2.
48
49	## Phase 2: Run Tests
50
51	1. Ensure `tests/package.json` exists in the user's project. If not, copy it:
52	```bash
53	cp ${CLAUDE_SKILL_DIR}/../../tests/package.json ./tests/package.json
54	```
55
56	2. Install dependencies if `tests/node_modules/` doesn't exist:
57	```bash
58	npm install --prefix tests
59	```
60
61	3. Run the test script in the background with a 100-minute timeout (6000000ms):
62	```bash
63	node ${CLAUDE_SKILL_DIR}/../../tests/run-tests.js --config-dir ./tests
64	```
65	Use `run_in_background: true` for this command. Save the returned task ID.
66
67	4. Wait 10 seconds, then check the background task output (non-blocking check).
68
69	5. Detect the authentication state from the output:
70
71	- If the output contains "Using cached token": Authentication succeeded automatically. Tell the user: "Authentication successful (cached credentials). Tests are running, this may take several minutes..."
72
73	- If the output contains "use a web browser to open the page": Extract the URL and device code from the message. Present this prominently to the user:
74
75	> Authentication Required
76	>
77	> Open your browser to: https://microsoft.com/devicelogin
78	> Enter the code: XXXXXXXXX (extract the actual code from the output)
79	>
80	> After signing in, the tests will continue automatically.
81
82	- If the output contains an error: Report the error to the user and stop.
83
84	- If the output is empty or incomplete: Wait another 10 seconds and check again (retry up to 3 times).
85
86	6. Wait for the background task to complete (blocking). The script polls every 20 seconds until all tests finish and downloads results as a CSV.
87
88	7. Read the final output to get the success rate and CSV filename.
89
90	8. Proceed to Phase 3.
91
92	## Phase 3: Analyze Results
93
94	1. Get the results: `Glob: tests/test-results-*.csv` — read the most recent CSV file (newest by modification time).
95
96	2. Parse the CSV columns:
97	\| Column \| Meaning \|
98	\|--------\|---------\|
99	\| Test Utterance \| The user message that was t

People who liked this also viewed

6 picks

Type