ai-test-orchestration

Name: ai-test-orchestration
Author: PramodDutta

182stars17forks

How to install ai-test-orchestration?

pramoddutta/qaskills/ai-test-orchestration

$npx -y skills add PramodDutta/qaskills --skill ai-test-orchestration

Installs into the current project.

Summary

AI-powered test orchestration skill covering intelligent test selection, risk-based test prioritization, flaky test management, test impact analysis, parallel execution optimization, and predictive test failure detection using machine learning.

Files · 1

View on GitHub

SKILL.md

1	# AI Test Orchestration Skill
2
3	You are an expert software engineer specializing in AI-powered test orchestration and intelligent test management. When the user asks you to implement, optimize, or debug test selection, prioritization, or parallel execution strategies, follow these detailed instructions.
4
5	## Core Principles
6
7	1. Test the changed code first -- Prioritize tests that cover recently modified files and functions.
8	2. Learn from history -- Use historical pass/fail data to predict which tests are likely to fail.
9	3. Quarantine, don't ignore -- Flaky tests should be isolated and tracked, not deleted or skipped.
10	4. Optimize for feedback speed -- Run the most likely-to-fail tests first so developers get fast signals.
11	5. Distribute intelligently -- Split test suites across parallel workers based on historical duration, not file count.
12	6. Measure and iterate -- Track metrics like time-to-first-failure, false positive rate, and test suite efficiency.
13	7. Fail fast, verify thoroughly -- Fast feedback on PR checks, comprehensive verification on merge.
14
15	## Project Structure
16
17	```
18	project/
19	src/
20	orchestrator/
21	test-selector.ts
22	risk-scorer.ts
23	impact-analyzer.ts
24	parallel-splitter.ts
25	flaky-detector.ts
26	prediction-model.ts
27	data/
28	test-history.ts
29	git-analysis.ts
30	coverage-map.ts
31	reporters/
32	orchestration-report.ts
33	metrics-collector.ts
34	config/
35	orchestration.config.ts
36	scripts/
37	collect-test-data.ts
38	train-model.py
39	analyze-flakiness.ts
40	tests/
41	orchestrator/
42	test-selector.test.ts
43	risk-scorer.test.ts
44	impact-analyzer.test.ts
45	```
46
47	## Intelligent Test Selection Based on Code Changes
48
49	```typescript
50	// src/orchestrator/test-selector.ts
51	import { execSync } from 'child_process';
52	import { CoverageMap } from '../data/coverage-map';
53
54	interface TestSelection {
55	mustRun: string[]; // Tests directly covering changed code
56	shouldRun: string[]; // Tests with transitive dependencies on changed code
57	canSkip: string[]; // Tests with no relation to changes
58	confidence: number; // 0-1, how confident we are in the selection
59	}
60
61	interface ChangedFile {
62	path: string;
63	additions: number;
64	deletions: number;
65	changedFunctions: string[];
66	}
67
68	export class TestSelector {
69	private coverageMap: CoverageMap;
70
71	constructor(coverageMap: CoverageMap) {
72	this.coverageMap = coverageMap;
73	}
74
75	async selectTests(baseBranch: string = 'main'): Promise<TestSelection> {
76	const changedFiles = this.getChangedFiles(baseBranch);
77	const allTests = this.coverageMap.getAllTests();
78	const mustRun = new Set<string>();
79	const shouldRun = new Set<string>();
80
81	for (const file of changedFiles) {
82	// Direct coverage: tests that execute lines in this file
83	const directTests = this.coverageMap.getTestsCoveringFile(file.path);
84	directTests.forEach((t) => mustRun.add(t));
85
86	// Function-level precision: only tests covering changed functions
87	if (file.changedFunctions.length > 0) {
88	for (const fn of file.changedFunctions) {
89	const fnTests = this.coverageMap.getTestsCoveringFunction(file.path, fn);
90	fnTests.forEach((t) => mustRun.add(t));
91	}
92	}
93
94	// Transitive dependencies: tests covering files that import changed file
95	const dependents = this.coverageMap.getDependentsOf(file.path);
96	for (const dep of dependents) {
97	const depTests = this.coverageMap.getTestsCoveringFile(dep);
98	depTests.forEach((t) => {
99	if (!mustRun.has(t)) shouldRun.add(t);
100	});
101	}
102	}
103
104	const canSkip = allTests.filter(
105	(t) => !mustRun.has(t) && !shouldRun.has(t)
106	);
107
108	const confidence = this.coverageMap.isComplete()
109	? 0.95
110	: 0.7; // Lower confidence if coverage data is stale
111
112	return {
113	mustRun: [...mustRun],
114	shouldRun: [...shouldRun],
115	canSkip,
116	confidence,
117	};
118	}
119
120	private getChangedFiles(baseBranch: string): ChangedFile[] {
121	const diffOutput = execSync(
122	`git diff --name-only --diff-filter=ACMR ${baseBranch}...HEAD`,
123	{ encoding: 'utf-8' }
124	).trim();
125
126	if (!diffOutput) return [];
127
128	return diffO

People who liked this also viewed

6 picks

Type

ai-test-orchestration

182stars17forks

View on GitHub

How to install ai-test-orchestration?

pramoddutta/qaskills/ai-test-orchestration

$npx -y skills add PramodDutta/qaskills --skill ai-test-orchestration

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AI Test Orchestration Skill
2
3	You are an expert software engineer specializing in AI-powered test orchestration and intelligent test management. When the user asks you to implement, optimize, or debug test selection, prioritization, or parallel execution strategies, follow these detailed instructions.
4
5	## Core Principles
6
7	1. Test the changed code first -- Prioritize tests that cover recently modified files and functions.
8	2. Learn from history -- Use historical pass/fail data to predict which tests are likely to fail.
9	3. Quarantine, don't ignore -- Flaky tests should be isolated and tracked, not deleted or skipped.
10	4. Optimize for feedback speed -- Run the most likely-to-fail tests first so developers get fast signals.
11	5. Distribute intelligently -- Split test suites across parallel workers based on historical duration, not file count.
12	6. Measure and iterate -- Track metrics like time-to-first-failure, false positive rate, and test suite efficiency.
13	7. Fail fast, verify thoroughly -- Fast feedback on PR checks, comprehensive verification on merge.
14
15	## Project Structure
16
17	```
18	project/
19	src/
20	orchestrator/
21	test-selector.ts
22	risk-scorer.ts
23	impact-analyzer.ts
24	parallel-splitter.ts
25	flaky-detector.ts
26	prediction-model.ts
27	data/
28	test-history.ts
29	git-analysis.ts
30	coverage-map.ts
31	reporters/
32	orchestration-report.ts
33	metrics-collector.ts
34	config/
35	orchestration.config.ts
36	scripts/
37	collect-test-data.ts
38	train-model.py
39	analyze-flakiness.ts
40	tests/
41	orchestrator/
42	test-selector.test.ts
43	risk-scorer.test.ts
44	impact-analyzer.test.ts
45	```
46
47	## Intelligent Test Selection Based on Code Changes
48
49	```typescript
50	// src/orchestrator/test-selector.ts
51	import { execSync } from 'child_process';
52	import { CoverageMap } from '../data/coverage-map';
53
54	interface TestSelection {
55	mustRun: string[]; // Tests directly covering changed code
56	shouldRun: string[]; // Tests with transitive dependencies on changed code
57	canSkip: string[]; // Tests with no relation to changes
58	confidence: number; // 0-1, how confident we are in the selection
59	}
60
61	interface ChangedFile {
62	path: string;
63	additions: number;
64	deletions: number;
65	changedFunctions: string[];
66	}
67
68	export class TestSelector {
69	private coverageMap: CoverageMap;
70
71	constructor(coverageMap: CoverageMap) {
72	this.coverageMap = coverageMap;
73	}
74
75	async selectTests(baseBranch: string = 'main'): Promise<TestSelection> {
76	const changedFiles = this.getChangedFiles(baseBranch);
77	const allTests = this.coverageMap.getAllTests();
78	const mustRun = new Set<string>();
79	const shouldRun = new Set<string>();
80
81	for (const file of changedFiles) {
82	// Direct coverage: tests that execute lines in this file
83	const directTests = this.coverageMap.getTestsCoveringFile(file.path);
84	directTests.forEach((t) => mustRun.add(t));
85
86	// Function-level precision: only tests covering changed functions
87	if (file.changedFunctions.length > 0) {
88	for (const fn of file.changedFunctions) {
89	const fnTests = this.coverageMap.getTestsCoveringFunction(file.path, fn);
90	fnTests.forEach((t) => mustRun.add(t));
91	}
92	}
93
94	// Transitive dependencies: tests covering files that import changed file
95	const dependents = this.coverageMap.getDependentsOf(file.path);
96	for (const dep of dependents) {
97	const depTests = this.coverageMap.getTestsCoveringFile(dep);
98	depTests.forEach((t) => {
99	if (!mustRun.has(t)) shouldRun.add(t);
100	});
101	}
102	}
103
104	const canSkip = allTests.filter(
105	(t) => !mustRun.has(t) && !shouldRun.has(t)
106	);
107
108	const confidence = this.coverageMap.isComplete()
109	? 0.95
110	: 0.7; // Lower confidence if coverage data is stale
111
112	return {
113	mustRun: [...mustRun],
114	shouldRun: [...shouldRun],
115	canSkip,
116	confidence,
117	};
118	}
119
120	private getChangedFiles(baseBranch: string): ChangedFile[] {
121	const diffOutput = execSync(
122	`git diff --name-only --diff-filter=ACMR ${baseBranch}...HEAD`,
123	{ encoding: 'utf-8' }
124	).trim();
125
126	if (!diffOutput) return [];
127
128	return diffO

People who liked this also viewed

6 picks

Type