ab-test-analysis

Name: ab-test-analysis
Author: phuryn

1.8kinstalls2.5kforks

How to install ab-test-analysis?

phuryn/pm-skills/ab-test-analysis

$npx -y skills add phuryn/pm-skills --skill ab-test-analysis

Installs into the current project.

Summary

Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results, checking if a test reached significance, interpreting split test data, or deciding whether to ship a

Files · 1

View on GitHub

SKILL.md

1	## A/B Test Analysis
2
3	Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
4
5	### Context
6
7	You are analyzing A/B test results for $ARGUMENTS.
8
9	If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
10
11	### Instructions
12
13	1. Understand the experiment:
14	- What was the hypothesis?
15	- What was changed (the variant)?
16	- What is the primary metric? Any guardrail metrics?
17	- How long did the test run?
18	- What is the traffic split?
19
20	2. Validate the test setup:
21	- Sample size: Is the sample large enough for the expected effect size?
22	- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
23	- Flag if the test is underpowered (<80% power)
24	- Duration: Did the test run for at least 1-2 full business cycles?
25	- Randomization: Any evidence of sample ratio mismatch (SRM)?
26	- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
27
28	3. Calculate statistical significance:
29	- Conversion rate for control and variant
30	- Relative lift: (variant - control) / control × 100
31	- p-value: Using a two-tailed z-test or chi-squared test
32	- Confidence interval: 95% CI for the difference
33	- Statistical significance: Is p < 0.05?
34	- Practical significance: Is the lift meaningful for the business?
35
36	If the user provides raw data, generate and run a Python script to calculate these.
37
38	4. Check guardrail metrics:
39	- Did any guardrail metrics (revenue, engagement, page load time) degrade?
40	- A winning primary metric with degraded guardrails may not be a true win
41
42	5. Interpret results:
43
44	\| Outcome \| Recommendation \|
45	\|---\|---\|
46	\| Significant positive lift, no guardrail issues \| Ship it — roll out to 100% \|
47	\| Significant positive lift, guardrail concerns \| Investigate — understand trade-offs before shipping \|
48	\| Not significant, positive trend \| Extend the test — need more data or larger effect \|
49	\| Not significant, flat \| Stop the test — no meaningful difference detected \|
50	\| Significant negative lift \| Don't ship — revert to control, analyze why \|
51
52	6. Provide the analysis summary:
53	```
54	## A/B Test Results: [Test Name]
55
56	Hypothesis: [What we expected]
57	Duration: [X days] \| Sample: [N control / M variant]
58
59	\| Metric \| Control \| Variant \| Lift \| p-value \| Significant? \|
60	\|---\|---\|---\|---\|---\|---\|
61	\| [Primary] \| X% \| Y% \| +Z% \| 0.0X \| Yes/No \|
62	\| [Guardrail] \| ... \| ... \| ... \| ... \| ... \|
63
64	Recommendation: [Ship / Extend / Stop / Investigate]
65	Reasoning: [Why]
66	Next steps: [What to do]
67	```
68
69	Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
70
71	---
72
73	### Further Reading
74
75	- [A/B Testing 101 + Examples](https://www.productcompass.pm/p/ab-testing-101-for-pms)
76	- [Testing Product Ideas: The Ultimate Validation Experiments Library](https://www.productcompass.pm/p/the-ultimate-experiments-library)
77	- [Are You Tracking the Right Metrics?](https://www.productcompass.pm/p/are-you-tracking-the-right-metrics)

People who liked this also viewed

6 picks

Type

ab-test-analysis

1.8kinstalls2.5kforks

View on GitHub

How to install ab-test-analysis?

phuryn/pm-skills/ab-test-analysis

$npx -y skills add phuryn/pm-skills --skill ab-test-analysis

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	## A/B Test Analysis
2
3	Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.
4
5	### Context
6
7	You are analyzing A/B test results for $ARGUMENTS.
8
9	If the user provides data files (CSV, Excel, or analytics exports), read and analyze them directly. Generate Python scripts for statistical calculations when needed.
10
11	### Instructions
12
13	1. Understand the experiment:
14	- What was the hypothesis?
15	- What was changed (the variant)?
16	- What is the primary metric? Any guardrail metrics?
17	- How long did the test run?
18	- What is the traffic split?
19
20	2. Validate the test setup:
21	- Sample size: Is the sample large enough for the expected effect size?
22	- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²
23	- Flag if the test is underpowered (<80% power)
24	- Duration: Did the test run for at least 1-2 full business cycles?
25	- Randomization: Any evidence of sample ratio mismatch (SRM)?
26	- Novelty/primacy effects: Was there enough time to wash out initial behavior changes?
27
28	3. Calculate statistical significance:
29	- Conversion rate for control and variant
30	- Relative lift: (variant - control) / control × 100
31	- p-value: Using a two-tailed z-test or chi-squared test
32	- Confidence interval: 95% CI for the difference
33	- Statistical significance: Is p < 0.05?
34	- Practical significance: Is the lift meaningful for the business?
35
36	If the user provides raw data, generate and run a Python script to calculate these.
37
38	4. Check guardrail metrics:
39	- Did any guardrail metrics (revenue, engagement, page load time) degrade?
40	- A winning primary metric with degraded guardrails may not be a true win
41
42	5. Interpret results:
43
44	\| Outcome \| Recommendation \|
45	\|---\|---\|
46	\| Significant positive lift, no guardrail issues \| Ship it — roll out to 100% \|
47	\| Significant positive lift, guardrail concerns \| Investigate — understand trade-offs before shipping \|
48	\| Not significant, positive trend \| Extend the test — need more data or larger effect \|
49	\| Not significant, flat \| Stop the test — no meaningful difference detected \|
50	\| Significant negative lift \| Don't ship — revert to control, analyze why \|
51
52	6. Provide the analysis summary:
53	```
54	## A/B Test Results: [Test Name]
55
56	Hypothesis: [What we expected]
57	Duration: [X days] \| Sample: [N control / M variant]
58
59	\| Metric \| Control \| Variant \| Lift \| p-value \| Significant? \|
60	\|---\|---\|---\|---\|---\|---\|
61	\| [Primary] \| X% \| Y% \| +Z% \| 0.0X \| Yes/No \|
62	\| [Guardrail] \| ... \| ... \| ... \| ... \| ... \|
63
64	Recommendation: [Ship / Extend / Stop / Investigate]
65	Reasoning: [Why]
66	Next steps: [What to do]
67	```
68
69	Think step by step. Save as markdown. Generate Python scripts for calculations if raw data is provided.
70
71	---
72
73	### Further Reading
74
75	- [A/B Testing 101 + Examples](https://www.productcompass.pm/p/ab-testing-101-for-pms)
76	- [Testing Product Ideas: The Ultimate Validation Experiments Library](https://www.productcompass.pm/p/the-ultimate-experiments-library)
77	- [Are You Tracking the Right Metrics?](https://www.productcompass.pm/p/are-you-tracking-the-right-metrics)

People who liked this also viewed

6 picks

Type