ab-testing

Name: ab-testing
Author: coreyhaines31

35kinstalls6.5kforks

How to install ab-testing?

coreyhaines31/marketingskills/ab-testing

$npx -y skills add coreyhaines31/marketingskills --skill ab-testing

Installs into the current project.

Summary

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test th

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
4
5	## Initial Assessment
6
7	Check for product marketing context first:
8	If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
9
10	Before designing a test, understand:
11
12	1. Test Context - What are you trying to improve? What change are you considering?
13	2. Current State - Baseline conversion rate? Current traffic volume?
14	3. Constraints - Technical complexity? Timeline? Tools available?
15
16	---
17
18	## Core Principles
19
20	### 1. Start with a Hypothesis
21	- Not just "let's see what happens"
22	- Specific prediction of outcome
23	- Based on reasoning or data
24
25	### 2. Test One Thing
26	- Single variable per test
27	- Otherwise you don't know what worked
28
29	### 3. Statistical Rigor
30	- Pre-determine sample size
31	- Don't peek and stop early
32	- Commit to the methodology
33
34	### 4. Measure What Matters
35	- Primary metric tied to business value
36	- Secondary metrics for context
37	- Guardrail metrics to prevent harm
38
39	---
40
41	## Hypothesis Framework
42
43	### Structure
44
45	```
46	Because [observation/data],
47	we believe [change]
48	will cause [expected outcome]
49	for [audience].
50	We'll know this is true when [metrics].
51	```
52
53	### Example
54
55	Weak: "Changing the button color might increase clicks."
56
57	Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
58
59	---
60
61	## Test Types
62
63	\| Type \| Description \| Traffic Needed \|
64	\|------\|-------------\|----------------\|
65	\| A/B \| Two versions, single change \| Moderate \|
66	\| A/B/n \| Multiple variants \| Higher \|
67	\| MVT \| Multiple changes in combinations \| Very high \|
68	\| Split URL \| Different URLs for variants \| Moderate \|
69
70	---
71
72	## Sample Size
73
74	### Quick Reference
75
76	\| Baseline \| 10% Lift \| 20% Lift \| 50% Lift \|
77	\|----------\|----------\|----------\|----------\|
78	\| 1% \| 150k/variant \| 39k/variant \| 6k/variant \|
79	\| 3% \| 47k/variant \| 12k/variant \| 2k/variant \|
80	\| 5% \| 27k/variant \| 7k/variant \| 1.2k/variant \|
81	\| 10% \| 12k/variant \| 3k/variant \| 550/variant \|
82
83	Calculators:
84	- [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html)
85	- [Optimizely's](https://www.optimizely.com/sample-size-calculator/)
86
87	For detailed sample size tables and duration calculations: See [references/sample-size-guide.md](references/sample-size-guide.md)
88
89	---
90
91	## Metrics Selection
92
93	### Primary Metric
94	- Single metric that matters most
95	- Directly tied to hypothesis
96	- What you'll use to call the test
97
98	### Secondary Metrics
99	- Support primary metric interpretation
100	- Explain why/how the change worked
101
102	### Guardrail Metrics
103	- Things that shouldn't get worse
104	- Stop test if significantly negative
105
106	### Example: Pricing Page Test
107	- Primary: Plan selection rate
108	- Secondary: Time on page, plan distribution
109	- Guardrail: Support tickets, refund rate
110
111	---
112
113	## Designing Variants
114
115	### What to Vary
116
117	\| Category \| Examples \|
118	\|----------\|----------\|
119	\| Headlines/Copy \| Message angle, value prop, specificity, tone \|
120	\| Visual Design \| Layout, color, images, hierarchy \|
121	\| CTA \| Button copy, size, placement, number \|
122	\| Content \| Information included, order, amount, social proof \|
123
124	### Best Practices
125	- Single, meaningful change
126	- Bold enough to make a difference
127	- True to the hypothesis
128
129	---
130
131	## Traffic Allocation
132
133	\| Approach \| Split \| When to Use \|
134	\|----------\|-------\|-------------\|
135	\| Standard \| 50/50 \| Default for A/B \|
136	\| Conservative \| 90/10, 80/20 \| Limit risk of bad variant \|
137	\| Ramping \| Start small, increase \| Technical risk mitigation \|
138
139	Considerations:
140	- Consistency: Users see same variant on return
141	- Balanced exposure across time of day/week
142
143	---
144
145	## Implementation
146
147	### Client-Side
148	- JavaScript modifies page after load
149	- Quick to implement, can cause f

People who liked this also viewed

6 picks

Type

ab-testing

35kinstalls6.5kforks

View on GitHub

How to install ab-testing?

coreyhaines31/marketingskills/ab-testing

$npx -y skills add coreyhaines31/marketingskills --skill ab-testing

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
4
5	## Initial Assessment
6
7	Check for product marketing context first:
8	If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
9
10	Before designing a test, understand:
11
12	1. Test Context - What are you trying to improve? What change are you considering?
13	2. Current State - Baseline conversion rate? Current traffic volume?
14	3. Constraints - Technical complexity? Timeline? Tools available?
15
16	---
17
18	## Core Principles
19
20	### 1. Start with a Hypothesis
21	- Not just "let's see what happens"
22	- Specific prediction of outcome
23	- Based on reasoning or data
24
25	### 2. Test One Thing
26	- Single variable per test
27	- Otherwise you don't know what worked
28
29	### 3. Statistical Rigor
30	- Pre-determine sample size
31	- Don't peek and stop early
32	- Commit to the methodology
33
34	### 4. Measure What Matters
35	- Primary metric tied to business value
36	- Secondary metrics for context
37	- Guardrail metrics to prevent harm
38
39	---
40
41	## Hypothesis Framework
42
43	### Structure
44
45	```
46	Because [observation/data],
47	we believe [change]
48	will cause [expected outcome]
49	for [audience].
50	We'll know this is true when [metrics].
51	```
52
53	### Example
54
55	Weak: "Changing the button color might increase clicks."
56
57	Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
58
59	---
60
61	## Test Types
62
63	\| Type \| Description \| Traffic Needed \|
64	\|------\|-------------\|----------------\|
65	\| A/B \| Two versions, single change \| Moderate \|
66	\| A/B/n \| Multiple variants \| Higher \|
67	\| MVT \| Multiple changes in combinations \| Very high \|
68	\| Split URL \| Different URLs for variants \| Moderate \|
69
70	---
71
72	## Sample Size
73
74	### Quick Reference
75
76	\| Baseline \| 10% Lift \| 20% Lift \| 50% Lift \|
77	\|----------\|----------\|----------\|----------\|
78	\| 1% \| 150k/variant \| 39k/variant \| 6k/variant \|
79	\| 3% \| 47k/variant \| 12k/variant \| 2k/variant \|
80	\| 5% \| 27k/variant \| 7k/variant \| 1.2k/variant \|
81	\| 10% \| 12k/variant \| 3k/variant \| 550/variant \|
82
83	Calculators:
84	- [Evan Miller's](https://www.evanmiller.org/ab-testing/sample-size.html)
85	- [Optimizely's](https://www.optimizely.com/sample-size-calculator/)
86
87	For detailed sample size tables and duration calculations: See [references/sample-size-guide.md](references/sample-size-guide.md)
88
89	---
90
91	## Metrics Selection
92
93	### Primary Metric
94	- Single metric that matters most
95	- Directly tied to hypothesis
96	- What you'll use to call the test
97
98	### Secondary Metrics
99	- Support primary metric interpretation
100	- Explain why/how the change worked
101
102	### Guardrail Metrics
103	- Things that shouldn't get worse
104	- Stop test if significantly negative
105
106	### Example: Pricing Page Test
107	- Primary: Plan selection rate
108	- Secondary: Time on page, plan distribution
109	- Guardrail: Support tickets, refund rate
110
111	---
112
113	## Designing Variants
114
115	### What to Vary
116
117	\| Category \| Examples \|
118	\|----------\|----------\|
119	\| Headlines/Copy \| Message angle, value prop, specificity, tone \|
120	\| Visual Design \| Layout, color, images, hierarchy \|
121	\| CTA \| Button copy, size, placement, number \|
122	\| Content \| Information included, order, amount, social proof \|
123
124	### Best Practices
125	- Single, meaningful change
126	- Bold enough to make a difference
127	- True to the hypothesis
128
129	---
130
131	## Traffic Allocation
132
133	\| Approach \| Split \| When to Use \|
134	\|----------\|-------\|-------------\|
135	\| Standard \| 50/50 \| Default for A/B \|
136	\| Conservative \| 90/10, 80/20 \| Limit risk of bad variant \|
137	\| Ramping \| Start small, increase \| Technical risk mitigation \|
138
139	Considerations:
140	- Consistency: Users see same variant on return
141	- Balanced exposure across time of day/week
142
143	---
144
145	## Implementation
146
147	### Client-Side
148	- JavaScript modifies page after load
149	- Quick to implement, can cause f

People who liked this also viewed

6 picks

Type