ab-test-setup

Name: ab-test-setup
Author: sickn33

44kstars6.5kforks

How to install ab-test-setup?

sickn33/agentic-awesome-skills/ab-test-setup

$npx -y skills add sickn33/agentic-awesome-skills --skill ab-test-setup

Installs into the current project.

Summary

Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness.

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	## 1️⃣ Purpose & Scope
4
5	Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
6
7	- Prevents "peeking"
8	- Enforces statistical power
9	- Blocks invalid hypotheses
10
11	---
12
13	## 2️⃣ Pre-Requisites
14
15	You must have:
16
17	- A clear user problem
18	- Access to an analytics source
19	- Roughly estimated traffic volume
20
21	### Hypothesis Quality Checklist
22
23	A valid hypothesis includes:
24
25	- Observation or evidence
26	- Single, specific change
27	- Directional expectation
28	- Defined audience
29	- Measurable success criteria
30
31	---
32
33	## 3️⃣ Hypothesis Lock (Hard Gate)
34
35	Before designing variants or metrics, you MUST:
36
37	- Present the final hypothesis
38	- Specify:
39	- Target audience
40	- Primary metric
41	- Expected direction of effect
42	- Minimum Detectable Effect (MDE)
43
44	Ask explicitly:
45
46	> “Is this the final hypothesis we are committing to for this test?”
47
48	Do NOT proceed until confirmed.
49
50	---
51
52	## 4️⃣ Assumptions & Validity Check (Mandatory)
53
54	Explicitly list assumptions about:
55
56	- Traffic stability
57	- User independence
58	- Metric reliability
59	- Randomization quality
60	- External factors (seasonality, campaigns, releases)
61
62	If assumptions are weak or violated:
63
64	- Warn the user
65	- Recommend delaying or redesigning the test
66
67	---
68
69	## 5️⃣ Test Type Selection
70
71	Choose the simplest valid test:
72
73	- A/B Test – single change, two variants
74	- A/B/n Test – multiple variants, higher traffic required
75	- Multivariate Test (MVT) – interaction effects, very high traffic
76	- Split URL Test – major structural changes
77
78	Default to A/B unless there is a clear reason otherwise.
79
80	---
81
82	## 6️⃣ Metrics Definition
83
84	#### Primary Metric (Mandatory)
85
86	- Single metric used to evaluate success
87	- Directly tied to the hypothesis
88	- Pre-defined and frozen before launch
89
90	#### Secondary Metrics
91
92	- Provide context
93	- Explain _why_ results occurred
94	- Must not override the primary metric
95
96	#### Guardrail Metrics
97
98	- Metrics that must not degrade
99	- Used to prevent harmful wins
100	- Trigger test stop if significantly negative
101
102	---
103
104	## 7️⃣ Sample Size & Duration
105
106	Define upfront:
107
108	- Baseline rate
109	- MDE
110	- Significance level (typically 95%)
111	- Statistical power (typically 80%)
112
113	Estimate:
114
115	- Required sample size per variant
116	- Expected test duration
117
118	Do NOT proceed without a realistic sample size estimate.
119
120	---
121
122	### Tracking Verification (Required before Gate 8)
123
124	Before entering the Execution Readiness Gate below, run through this checklist to make "Tracking is verified" mean something concrete:
125
126	1. Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug page, and confirm it lands in your analytics destination within 30 seconds.
127	2. Variant attribution: Verify that the variant assignment ID is attached to every fired event — not just the entry event. Use your analytics' raw event view to compare a sample of 5+ events per variant.
128	3. De-duplication: Confirm that a user reloading the page does not cause double-counted events. If your stack uses client-side de-duping, the variant ID must be part of the dedup key.
129	4. Sample randomization: Pull the first 100 assignment records from your assignment table; the variant split should be within ±5% of the configured allocation.
130	5. Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.
131
132	If any of the above fails, stop and resolve it before Gate 8.
133
134	---
135
136	## 8️⃣ Execution Readiness Gate (Hard Stop)
137
138	You may proceed to implementation only if all are true:
139
140	- Hypothesis is locked
141	- Primary metric is frozen
142	- Sample size is calculated
143	- Test duration is defined
144	- Guardrails are set
145	- Tracking is verified
146
147	If any item is missing, stop and resolve it.
148
149	---
150
151	## Running the Test
152
153	### During the Test
154
155	DO:
156
157	- Monitor technical health
158	- Document external factors
159
160	DO NOT:
161
162	- Stop early due to “good-looking” results
163	- Change variants mid-test
164	- Add new traffic sources
165	- Redefine success criteria
166
167	---
168
169	## Analyzing Results
170
171	### Analysis Discipline
172
173	When interpreting results:
174
175	- Do NOT generalize beyond the tested population
176	- Do NOT claim causality beyond the tested change
177	- Do NOT override guardrail failures
178	- Separate statistical significance from business judgment
179
180	### Interpretation Outcomes
181
182	\| Result \| Action \|
183	\| -------------------- \| -------------------------------------- \|
184	\| Significant positive \| Consider rollout \|
185	\| Significant negative \| Reject variant, document learning \|
186	\| Inconclusive \| Consider more traffic or bolder change \|
187	\| Guardrail failure \| Do not ship, even if primary wins \|
188
189	---
190
191	## Documentation & Learning
192
193	###

People who liked this also viewed

6 picks

Type

tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026501k181k
test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026175k259k
qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026160k181k
verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026153k259k
webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026120k163k
playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 202664k338

ab-test-setup

44kstars6.5kforks

View on GitHub

How to install ab-test-setup?

sickn33/agentic-awesome-skills/ab-test-setup

$npx -y skills add sickn33/agentic-awesome-skills --skill ab-test-setup

Installs into the current project.

Summary

Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness.

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	## 1️⃣ Purpose & Scope
4
5	Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
6
7	- Prevents "peeking"
8	- Enforces statistical power
9	- Blocks invalid hypotheses
10
11	---
12
13	## 2️⃣ Pre-Requisites
14
15	You must have:
16
17	- A clear user problem
18	- Access to an analytics source
19	- Roughly estimated traffic volume
20
21	### Hypothesis Quality Checklist
22
23	A valid hypothesis includes:
24
25	- Observation or evidence
26	- Single, specific change
27	- Directional expectation
28	- Defined audience
29	- Measurable success criteria
30
31	---
32
33	## 3️⃣ Hypothesis Lock (Hard Gate)
34
35	Before designing variants or metrics, you MUST:
36
37	- Present the final hypothesis
38	- Specify:
39	- Target audience
40	- Primary metric
41	- Expected direction of effect
42	- Minimum Detectable Effect (MDE)
43
44	Ask explicitly:
45
46	> “Is this the final hypothesis we are committing to for this test?”
47
48	Do NOT proceed until confirmed.
49
50	---
51
52	## 4️⃣ Assumptions & Validity Check (Mandatory)
53
54	Explicitly list assumptions about:
55
56	- Traffic stability
57	- User independence
58	- Metric reliability
59	- Randomization quality
60	- External factors (seasonality, campaigns, releases)
61
62	If assumptions are weak or violated:
63
64	- Warn the user
65	- Recommend delaying or redesigning the test
66
67	---
68
69	## 5️⃣ Test Type Selection
70
71	Choose the simplest valid test:
72
73	- A/B Test – single change, two variants
74	- A/B/n Test – multiple variants, higher traffic required
75	- Multivariate Test (MVT) – interaction effects, very high traffic
76	- Split URL Test – major structural changes
77
78	Default to A/B unless there is a clear reason otherwise.
79
80	---
81
82	## 6️⃣ Metrics Definition
83
84	#### Primary Metric (Mandatory)
85
86	- Single metric used to evaluate success
87	- Directly tied to the hypothesis
88	- Pre-defined and frozen before launch
89
90	#### Secondary Metrics
91
92	- Provide context
93	- Explain _why_ results occurred
94	- Must not override the primary metric
95
96	#### Guardrail Metrics
97
98	- Metrics that must not degrade
99	- Used to prevent harmful wins
100	- Trigger test stop if significantly negative
101
102	---
103
104	## 7️⃣ Sample Size & Duration
105
106	Define upfront:
107
108	- Baseline rate
109	- MDE
110	- Significance level (typically 95%)
111	- Statistical power (typically 80%)
112
113	Estimate:
114
115	- Required sample size per variant
116	- Expected test duration
117
118	Do NOT proceed without a realistic sample size estimate.
119
120	---
121
122	### Tracking Verification (Required before Gate 8)
123
124	Before entering the Execution Readiness Gate below, run through this checklist to make "Tracking is verified" mean something concrete:
125
126	1. Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug page, and confirm it lands in your analytics destination within 30 seconds.
127	2. Variant attribution: Verify that the variant assignment ID is attached to every fired event — not just the entry event. Use your analytics' raw event view to compare a sample of 5+ events per variant.
128	3. De-duplication: Confirm that a user reloading the page does not cause double-counted events. If your stack uses client-side de-duping, the variant ID must be part of the dedup key.
129	4. Sample randomization: Pull the first 100 assignment records from your assignment table; the variant split should be within ±5% of the configured allocation.
130	5. Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.
131
132	If any of the above fails, stop and resolve it before Gate 8.
133
134	---
135
136	## 8️⃣ Execution Readiness Gate (Hard Stop)
137
138	You may proceed to implementation only if all are true:
139
140	- Hypothesis is locked
141	- Primary metric is frozen
142	- Sample size is calculated
143	- Test duration is defined
144	- Guardrails are set
145	- Tracking is verified
146
147	If any item is missing, stop and resolve it.
148
149	---
150
151	## Running the Test
152
153	### During the Test
154
155	DO:
156
157	- Monitor technical health
158	- Document external factors
159
160	DO NOT:
161
162	- Stop early due to “good-looking” results
163	- Change variants mid-test
164	- Add new traffic sources
165	- Redefine success criteria
166
167	---
168
169	## Analyzing Results
170
171	### Analysis Discipline
172
173	When interpreting results:
174
175	- Do NOT generalize beyond the tested population
176	- Do NOT claim causality beyond the tested change
177	- Do NOT override guardrail failures
178	- Separate statistical significance from business judgment
179
180	### Interpretation Outcomes
181
182	\| Result \| Action \|
183	\| -------------------- \| -------------------------------------- \|
184	\| Significant positive \| Consider rollout \|
185	\| Significant negative \| Reject variant, document learning \|
186	\| Inconclusive \| Consider more traffic or bolder change \|
187	\| Guardrail failure \| Do not ship, even if primary wins \|
188
189	---
190
191	## Documentation & Learning
192
193	###

People who liked this also viewed

6 picks

Type