regression-testing

Name: regression-testing
Author: Productfculty-aipm

70stars33forks

How to install regression-testing?

productfculty-aipm/pm-copilot-by-product-faculty/regression-testing

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill regression-testing

Installs into the current project.

Summary

Use this skill when the user asks to "prevent regressions in AI quality", "regression testing for AI", "how do I know if a prompt change broke something", "before/after evaluation for model changes", "catch quality regressions", or wants to set up a process that catches when a mo

Files · 1

View on GitHub

SKILL.md

1	# AI Regression Testing
2
3	You are setting up a regression testing framework for an AI feature — a systematic process that catches quality degradations caused by model changes, prompt changes, or data/context changes before they reach users.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), software testing principles applied to AI.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being protected. Read the eval suite design if available — regression tests are a subset of the broader eval suite, focused on the specific failure modes the team has already identified and fixed.
10
11	## Step 2 — What Triggers a Regression?
12
13	AI regressions can be caused by:
14	- Model updates: The underlying LLM changes (e.g., Claude version update)
15	- Prompt changes: The system prompt or few-shot examples are modified
16	- Context changes: The data passed to the model (retrieved documents, user context) changes
17	- Tool/API changes: External tools the model calls change their behavior
18	- Distribution shift: The types of inputs coming from users have changed over time
19
20	The regression test suite should catch all of these, not just obvious changes.
21
22	## Step 3 — Building the Regression Test Set
23
24	The regression test set is a curated collection of (input, expected behavior) pairs. "Expected behavior" means the output should PASS a specific eval.
25
26	Sources for the test set:
27	1. Past failures that were fixed: If you fixed a failure in v1.0, add a test that would have caught it. This prevents it from recurring silently.
28	2. Edge cases from error analysis: The failure categories identified in error analysis become test cases.
29	3. Representative happy path examples: The most common, important use cases should have at least one test each.
30	4. Boundary cases: Inputs that are at the edge of the system's capability or scope.
31
32	Size guidance:
33	- Minimum viable: 20–30 carefully chosen test cases (covers major failure categories)
34	- Good: 50–100 test cases (covers failure categories + representative happy paths)
35	- Comprehensive: 200+ test cases (needed for high-stakes AI features)
36
37	## Step 4 — Regression Test Execution
38
39	When to run:
40	- On every PR that changes the system prompt, model version, or retrieval pipeline
41	- On every scheduled model update (when the underlying model is upgraded)
42	- Weekly (to catch silent regressions from distribution shift)
43
44	Pass/fail definition:
45	The test suite passes if: (1) every individual test case passes its specific eval, AND (2) the aggregate pass rate doesn't drop by more than [threshold]% from the baseline.
46
47	Set the threshold based on the feature's criticality:
48	- Core feature: < 2% regression acceptable
49	- Secondary feature: < 5% regression acceptable
50	- Experimental feature: < 10% regression acceptable
51
52	## Step 5 — Regression Report Structure
53
54	When a regression is detected, produce a report:
55
56	What changed: Which eval(s) failed? What does the failure pattern look like?
57
58	Affected input types: Are regressions concentrated on certain types of inputs (short inputs, specific user segments, specific task types)?
59
60	Severity: How many test cases failed? What's the regression % vs. baseline?
61
62	Root cause hypothesis: What change (model, prompt, context) most likely caused this?
63
64	Rollback recommendation: Should the change be reverted immediately, or is this a degradation that can be fixed forward?
65
66	Fix plan: If fixing forward, what changes to the prompt or system would address the regression?
67
68	## Step 6 — CI/CD Integration
69
70	Connect regression tests to the deployment pipeline:
71
72	```python
73	# Pseudocode: regression gate in deployment pipeline
74	def run_regression_gate(eval_suite, test_cases, baseline_pass_rate, threshold=0.02):
75	results = [run_eval(test_case, eval_suite) for test_case in test_cases]
76	current_pass_rate = sum(1 for r in results if r.passed) / len(results)
77	regression = baseline_pass_rate - current_pass_rate
78
79	if regression > threshold:
80	raise DeploymentBlockedError(
81	f"Regression detected: {regression:.1%} quality drop vs. baseline. "
82	f"Blocking deployment. Review failing cases: {[r for r in results if not r.passed]}"
83	)
84
85	return {"pass_rate": current_pass_rate, "regression": regression, "status": "PASS"}
86	```
87
88	## Step 7 — Regression Triage Process
89
90	When a regression is flagged:
91
92	1. Identify which test cases failed (which failure categories?)
93	2. Compare failing outputs to the passing outputs from before the change
94	3. Determine root cause: i

People who liked this also viewed

6 picks

Type

regression-testing

70stars33forks

View on GitHub

How to install regression-testing?

productfculty-aipm/pm-copilot-by-product-faculty/regression-testing

$npx -y skills add Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill regression-testing

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# AI Regression Testing
2
3	You are setting up a regression testing framework for an AI feature — a systematic process that catches quality degradations caused by model changes, prompt changes, or data/context changes before they reach users.
4
5	Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), software testing principles applied to AI.
6
7	## Step 1 — Load Context
8
9	Read `memory/user-profile.md` for the AI feature being protected. Read the eval suite design if available — regression tests are a subset of the broader eval suite, focused on the specific failure modes the team has already identified and fixed.
10
11	## Step 2 — What Triggers a Regression?
12
13	AI regressions can be caused by:
14	- Model updates: The underlying LLM changes (e.g., Claude version update)
15	- Prompt changes: The system prompt or few-shot examples are modified
16	- Context changes: The data passed to the model (retrieved documents, user context) changes
17	- Tool/API changes: External tools the model calls change their behavior
18	- Distribution shift: The types of inputs coming from users have changed over time
19
20	The regression test suite should catch all of these, not just obvious changes.
21
22	## Step 3 — Building the Regression Test Set
23
24	The regression test set is a curated collection of (input, expected behavior) pairs. "Expected behavior" means the output should PASS a specific eval.
25
26	Sources for the test set:
27	1. Past failures that were fixed: If you fixed a failure in v1.0, add a test that would have caught it. This prevents it from recurring silently.
28	2. Edge cases from error analysis: The failure categories identified in error analysis become test cases.
29	3. Representative happy path examples: The most common, important use cases should have at least one test each.
30	4. Boundary cases: Inputs that are at the edge of the system's capability or scope.
31
32	Size guidance:
33	- Minimum viable: 20–30 carefully chosen test cases (covers major failure categories)
34	- Good: 50–100 test cases (covers failure categories + representative happy paths)
35	- Comprehensive: 200+ test cases (needed for high-stakes AI features)
36
37	## Step 4 — Regression Test Execution
38
39	When to run:
40	- On every PR that changes the system prompt, model version, or retrieval pipeline
41	- On every scheduled model update (when the underlying model is upgraded)
42	- Weekly (to catch silent regressions from distribution shift)
43
44	Pass/fail definition:
45	The test suite passes if: (1) every individual test case passes its specific eval, AND (2) the aggregate pass rate doesn't drop by more than [threshold]% from the baseline.
46
47	Set the threshold based on the feature's criticality:
48	- Core feature: < 2% regression acceptable
49	- Secondary feature: < 5% regression acceptable
50	- Experimental feature: < 10% regression acceptable
51
52	## Step 5 — Regression Report Structure
53
54	When a regression is detected, produce a report:
55
56	What changed: Which eval(s) failed? What does the failure pattern look like?
57
58	Affected input types: Are regressions concentrated on certain types of inputs (short inputs, specific user segments, specific task types)?
59
60	Severity: How many test cases failed? What's the regression % vs. baseline?
61
62	Root cause hypothesis: What change (model, prompt, context) most likely caused this?
63
64	Rollback recommendation: Should the change be reverted immediately, or is this a degradation that can be fixed forward?
65
66	Fix plan: If fixing forward, what changes to the prompt or system would address the regression?
67
68	## Step 6 — CI/CD Integration
69
70	Connect regression tests to the deployment pipeline:
71
72	```python
73	# Pseudocode: regression gate in deployment pipeline
74	def run_regression_gate(eval_suite, test_cases, baseline_pass_rate, threshold=0.02):
75	results = [run_eval(test_case, eval_suite) for test_case in test_cases]
76	current_pass_rate = sum(1 for r in results if r.passed) / len(results)
77	regression = baseline_pass_rate - current_pass_rate
78
79	if regression > threshold:
80	raise DeploymentBlockedError(
81	f"Regression detected: {regression:.1%} quality drop vs. baseline. "
82	f"Blocking deployment. Review failing cases: {[r for r in results if not r.passed]}"
83	)
84
85	return {"pass_rate": current_pass_rate, "regression": regression, "status": "PASS"}
86	```
87
88	## Step 7 — Regression Triage Process
89
90	When a regression is flagged:
91
92	1. Identify which test cases failed (which failure categories?)
93	2. Compare failing outputs to the passing outputs from before the change
94	3. Determine root cause: i

People who liked this also viewed

6 picks

Type