ab-test-setup

Name: ab-test-setup
Author: aitytech

26installs73forks

How to install ab-test-setup?

aitytech/agentkits-marketing/ab-test-setup

$npx -y skills add aitytech/agentkits-marketing --skill ab-test-setup

Installs into the current project.

Summary

When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
4
5	## Initial Assessment
6
7	Before designing a test, understand:
8
9	1. Test Context
10	- What are you trying to improve?
11	- What change are you considering?
12	- What made you want to test this?
13
14	2. Current State
15	- Baseline conversion rate?
16	- Current traffic volume?
17	- Any historical test data?
18
19	3. Constraints
20	- Technical implementation complexity?
21	- Timeline requirements?
22	- Tools available?
23
24	---
25
26	## Core Principles
27
28	### 1. Start with a Hypothesis
29	- Not just "let's see what happens"
30	- Specific prediction of outcome
31	- Based on reasoning or data
32
33	### 2. Test One Thing
34	- Single variable per test
35	- Otherwise you don't know what worked
36	- Save MVT for later
37
38	### 3. Statistical Rigor
39	- Pre-determine sample size
40	- Don't peek and stop early
41	- Commit to the methodology
42
43	### 4. Measure What Matters
44	- Primary metric tied to business value
45	- Secondary metrics for context
46	- Guardrail metrics to prevent harm
47
48	---
49
50	## Hypothesis Framework
51
52	### Structure
53
54	```
55	Because [observation/data],
56	we believe [change]
57	will cause [expected outcome]
58	for [audience].
59	We'll know this is true when [metrics].
60	```
61
62	### Examples
63
64	Weak hypothesis:
65	"Changing the button color might increase clicks."
66
67	Strong hypothesis:
68	"Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
69
70	### Good Hypotheses Include
71
72	- Observation: What prompted this idea
73	- Change: Specific modification
74	- Effect: Expected outcome and direction
75	- Audience: Who this applies to
76	- Metric: How you'll measure success
77
78	---
79
80	## Test Types
81
82	### A/B Test (Split Test)
83	- Two versions: Control (A) vs. Variant (B)
84	- Single change between versions
85	- Most common, easiest to analyze
86
87	### A/B/n Test
88	- Multiple variants (A vs. B vs. C...)
89	- Requires more traffic
90	- Good for testing several options
91
92	### Multivariate Test (MVT)
93	- Multiple changes in combinations
94	- Tests interactions between changes
95	- Requires significantly more traffic
96	- Complex analysis
97
98	### Split URL Test
99	- Different URLs for variants
100	- Good for major page changes
101	- Easier implementation sometimes
102
103	---
104
105	## Sample Size Calculation
106
107	### Inputs Needed
108
109	1. Baseline conversion rate: Your current rate
110	2. Minimum detectable effect (MDE): Smallest change worth detecting
111	3. Statistical significance level: Usually 95%
112	4. Statistical power: Usually 80%
113
114	### Quick Reference
115
116	\| Baseline Rate \| 10% Lift \| 20% Lift \| 50% Lift \|
117	\|---------------\|----------\|----------\|----------\|
118	\| 1% \| 150k/variant \| 39k/variant \| 6k/variant \|
119	\| 3% \| 47k/variant \| 12k/variant \| 2k/variant \|
120	\| 5% \| 27k/variant \| 7k/variant \| 1.2k/variant \|
121	\| 10% \| 12k/variant \| 3k/variant \| 550/variant \|
122
123	### Formula Resources
124	- Evan Miller's calculator: https://www.evanmiller.org/ab-testing/sample-size.html
125	- Optimizely's calculator: https://www.optimizely.com/sample-size-calculator/
126
127	### Test Duration
128
129	```
130	Duration = Sample size needed per variant × Number of variants
131	───────────────────────────────────────────────────
132	Daily traffic to test page × Conversion rate
133	```
134
135	Minimum: 1-2 business cycles (usually 1-2 weeks)
136	Maximum: Avoid running too long (novelty effects, external factors)
137
138	---
139
140	## Metrics Selection
141
142	### Primary Metric
143	- Single metric that matters most
144	- Directly tied to hypothesis
145	- What you'll use to call the test
146
147	### Secondary Metrics
148	- Support primary metric interpretation
149	- Explain why/how the change worked
150	- Help understand user behavior
151
152	### Guardrail Metrics
153	- Things that shouldn't get worse
154	- Revenue, retention, satisfaction
155	- Stop test if significantly negative
156
157	### Metric Examples by Test Type
158
159	Homepage CTA test:
160	- Primary: CTA click-through rate
161	- Secondary: Time to click, scroll depth
162	- Guardrail: Bounce rate, downstream conversion
163
164	Pricing page test:
165	- Primary:

People who liked this also viewed

6 picks

Type

ab-test-setup

26installs73forks

View on GitHub

How to install ab-test-setup?

aitytech/agentkits-marketing/ab-test-setup

$npx -y skills add aitytech/agentkits-marketing --skill ab-test-setup

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
4
5	## Initial Assessment
6
7	Before designing a test, understand:
8
9	1. Test Context
10	- What are you trying to improve?
11	- What change are you considering?
12	- What made you want to test this?
13
14	2. Current State
15	- Baseline conversion rate?
16	- Current traffic volume?
17	- Any historical test data?
18
19	3. Constraints
20	- Technical implementation complexity?
21	- Timeline requirements?
22	- Tools available?
23
24	---
25
26	## Core Principles
27
28	### 1. Start with a Hypothesis
29	- Not just "let's see what happens"
30	- Specific prediction of outcome
31	- Based on reasoning or data
32
33	### 2. Test One Thing
34	- Single variable per test
35	- Otherwise you don't know what worked
36	- Save MVT for later
37
38	### 3. Statistical Rigor
39	- Pre-determine sample size
40	- Don't peek and stop early
41	- Commit to the methodology
42
43	### 4. Measure What Matters
44	- Primary metric tied to business value
45	- Secondary metrics for context
46	- Guardrail metrics to prevent harm
47
48	---
49
50	## Hypothesis Framework
51
52	### Structure
53
54	```
55	Because [observation/data],
56	we believe [change]
57	will cause [expected outcome]
58	for [audience].
59	We'll know this is true when [metrics].
60	```
61
62	### Examples
63
64	Weak hypothesis:
65	"Changing the button color might increase clicks."
66
67	Strong hypothesis:
68	"Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
69
70	### Good Hypotheses Include
71
72	- Observation: What prompted this idea
73	- Change: Specific modification
74	- Effect: Expected outcome and direction
75	- Audience: Who this applies to
76	- Metric: How you'll measure success
77
78	---
79
80	## Test Types
81
82	### A/B Test (Split Test)
83	- Two versions: Control (A) vs. Variant (B)
84	- Single change between versions
85	- Most common, easiest to analyze
86
87	### A/B/n Test
88	- Multiple variants (A vs. B vs. C...)
89	- Requires more traffic
90	- Good for testing several options
91
92	### Multivariate Test (MVT)
93	- Multiple changes in combinations
94	- Tests interactions between changes
95	- Requires significantly more traffic
96	- Complex analysis
97
98	### Split URL Test
99	- Different URLs for variants
100	- Good for major page changes
101	- Easier implementation sometimes
102
103	---
104
105	## Sample Size Calculation
106
107	### Inputs Needed
108
109	1. Baseline conversion rate: Your current rate
110	2. Minimum detectable effect (MDE): Smallest change worth detecting
111	3. Statistical significance level: Usually 95%
112	4. Statistical power: Usually 80%
113
114	### Quick Reference
115
116	\| Baseline Rate \| 10% Lift \| 20% Lift \| 50% Lift \|
117	\|---------------\|----------\|----------\|----------\|
118	\| 1% \| 150k/variant \| 39k/variant \| 6k/variant \|
119	\| 3% \| 47k/variant \| 12k/variant \| 2k/variant \|
120	\| 5% \| 27k/variant \| 7k/variant \| 1.2k/variant \|
121	\| 10% \| 12k/variant \| 3k/variant \| 550/variant \|
122
123	### Formula Resources
124	- Evan Miller's calculator: https://www.evanmiller.org/ab-testing/sample-size.html
125	- Optimizely's calculator: https://www.optimizely.com/sample-size-calculator/
126
127	### Test Duration
128
129	```
130	Duration = Sample size needed per variant × Number of variants
131	───────────────────────────────────────────────────
132	Daily traffic to test page × Conversion rate
133	```
134
135	Minimum: 1-2 business cycles (usually 1-2 weeks)
136	Maximum: Avoid running too long (novelty effects, external factors)
137
138	---
139
140	## Metrics Selection
141
142	### Primary Metric
143	- Single metric that matters most
144	- Directly tied to hypothesis
145	- What you'll use to call the test
146
147	### Secondary Metrics
148	- Support primary metric interpretation
149	- Explain why/how the change worked
150	- Help understand user behavior
151
152	### Guardrail Metrics
153	- Things that shouldn't get worse
154	- Revenue, retention, satisfaction
155	- Stop test if significantly negative
156
157	### Metric Examples by Test Type
158
159	Homepage CTA test:
160	- Primary: CTA click-through rate
161	- Secondary: Time to click, scroll depth
162	- Guardrail: Bounce rate, downstream conversion
163
164	Pricing page test:
165	- Primary:

People who liked this also viewed

6 picks

Type