ab-test-setup

Name: ab-test-setup
Author: SamarthaKV29

64stars12forks

How to install ab-test-setup?

samarthakv29/antigravity-god-mode/ab-test-setup

$npx -y skills add SamarthaKV29/antigravity-god-mode --skill ab-test-setup

Installs into the current project.

Summary

Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness.

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	## 1️⃣ Purpose & Scope
4
5	Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
6
7	- Prevents "peeking"
8	- Enforces statistical power
9	- Blocks invalid hypotheses
10
11	---
12
13	## 2️⃣ Pre-Requisites
14
15	You must have:
16
17	- A clear user problem
18	- Access to an analytics source
19	- Roughly estimated traffic volume
20
21	### Hypothesis Quality Checklist
22
23	A valid hypothesis includes:
24
25	- Observation or evidence
26	- Single, specific change
27	- Directional expectation
28	- Defined audience
29	- Measurable success criteria
30
31	---
32
33	### 3️⃣ Hypothesis Lock (Hard Gate)
34
35	Before designing variants or metrics, you MUST:
36
37	- Present the final hypothesis
38	- Specify:
39	- Target audience
40	- Primary metric
41	- Expected direction of effect
42	- Minimum Detectable Effect (MDE)
43
44	Ask explicitly:
45
46	> “Is this the final hypothesis we are committing to for this test?”
47
48	Do NOT proceed until confirmed.
49
50	---
51
52	### 4️⃣ Assumptions & Validity Check (Mandatory)
53
54	Explicitly list assumptions about:
55
56	- Traffic stability
57	- User independence
58	- Metric reliability
59	- Randomization quality
60	- External factors (seasonality, campaigns, releases)
61
62	If assumptions are weak or violated:
63
64	- Warn the user
65	- Recommend delaying or redesigning the test
66
67	---
68
69	### 5️⃣ Test Type Selection
70
71	Choose the simplest valid test:
72
73	- A/B Test – single change, two variants
74	- A/B/n Test – multiple variants, higher traffic required
75	- Multivariate Test (MVT) – interaction effects, very high traffic
76	- Split URL Test – major structural changes
77
78	Default to A/B unless there is a clear reason otherwise.
79
80	---
81
82	### 6️⃣ Metrics Definition
83
84	#### Primary Metric (Mandatory)
85
86	- Single metric used to evaluate success
87	- Directly tied to the hypothesis
88	- Pre-defined and frozen before launch
89
90	#### Secondary Metrics
91
92	- Provide context
93	- Explain _why_ results occurred
94	- Must not override the primary metric
95
96	#### Guardrail Metrics
97
98	- Metrics that must not degrade
99	- Used to prevent harmful wins
100	- Trigger test stop if significantly negative
101
102	---
103
104	### 7️⃣ Sample Size & Duration
105
106	Define upfront:
107
108	- Baseline rate
109	- MDE
110	- Significance level (typically 95%)
111	- Statistical power (typically 80%)
112
113	Estimate:
114
115	- Required sample size per variant
116	- Expected test duration
117
118	Do NOT proceed without a realistic sample size estimate.
119
120	---
121
122	### 8️⃣ Execution Readiness Gate (Hard Stop)
123
124	You may proceed to implementation only if all are true:
125
126	- Hypothesis is locked
127	- Primary metric is frozen
128	- Sample size is calculated
129	- Test duration is defined
130	- Guardrails are set
131	- Tracking is verified
132
133	If any item is missing, stop and resolve it.
134
135	---
136
137	## Running the Test
138
139	### During the Test
140
141	DO:
142
143	- Monitor technical health
144	- Document external factors
145
146	DO NOT:
147
148	- Stop early due to “good-looking” results
149	- Change variants mid-test
150	- Add new traffic sources
151	- Redefine success criteria
152
153	---
154
155	## Analyzing Results
156
157	### Analysis Discipline
158
159	When interpreting results:
160
161	- Do NOT generalize beyond the tested population
162	- Do NOT claim causality beyond the tested change
163	- Do NOT override guardrail failures
164	- Separate statistical significance from business judgment
165
166	### Interpretation Outcomes
167
168	\| Result \| Action \|
169	\| -------------------- \| -------------------------------------- \|
170	\| Significant positive \| Consider rollout \|
171	\| Significant negative \| Reject variant, document learning \|
172	\| Inconclusive \| Consider more traffic or bolder change \|
173	\| Guardrail failure \| Do not ship, even if primary wins \|
174
175	---
176
177	## Documentation & Learning
178
179	### Test Record (Mandatory)
180
181	Document:
182
183	- Hypothesis
184	- Variants
185	- Metrics
186	- Sample size vs achieved
187	- Results
188	- Decision
189	- Learnings
190	- Follow-up ideas
191
192	Store records in a shared, searchable location to avoid repeated failures.
193
194	---
195
196	## Refusal Conditions (Safety)
197
198	Refuse to proceed if:
199
200	- Baseline rate is unknown and cannot be estimated
201	- Traffic is insufficient to detect the MDE
202	- Primary metric is undefined
203	- Multiple variables are changed without proper design
204	- Hypothesis cannot be clearly stated
205
206	Explain why and recommend next steps.
207
208	---
209
210	## Key Principles (Non-Negotiable)
211
212	- One hypothesis per test
213	- One primary metric
214	- Commit before launch
215	- No peeking
216	- Learning over winning
217	- Statistical rigor first
218
219	---
220
221	## Final Reminder
222
223	A/B testing is not about proving ideas right.
224	It is about learning the truth with confidence.
225
226	If you feel tempted to rush, simplify, or “just try it” —
227	that is the signal to slow down and re-check the design.

People who liked this also viewed

6 picks

Type

tddSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026501k181k
test-driven-developmentUse when implementing any feature or bugfix, before writing implementation codeSkillsJul 2026175k259k
qaSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026160k181k
verification-before-completionUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidenceSkillsJul 2026153k259k
webapp-testingToolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser lSkillsJul 2026120k163k
playwright-best-practicesUse when writing Playwright tests, fixing flaky tests, debugging failures, implementing Page Object Model, configuring CI/CD, optimizing performance, mocking APIs, handling authentication or OAuth, teSkillsJul 202664k338

ab-test-setup

64stars12forks

View on GitHub

How to install ab-test-setup?

samarthakv29/antigravity-god-mode/ab-test-setup

$npx -y skills add SamarthaKV29/antigravity-god-mode --skill ab-test-setup

Installs into the current project.

Summary

Structured guide for setting up A/B tests with mandatory gates for hypothesis, metrics, and execution readiness.

Files · 1

View on GitHub

SKILL.md

1	# A/B Test Setup
2
3	## 1️⃣ Purpose & Scope
4
5	Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
6
7	- Prevents "peeking"
8	- Enforces statistical power
9	- Blocks invalid hypotheses
10
11	---
12
13	## 2️⃣ Pre-Requisites
14
15	You must have:
16
17	- A clear user problem
18	- Access to an analytics source
19	- Roughly estimated traffic volume
20
21	### Hypothesis Quality Checklist
22
23	A valid hypothesis includes:
24
25	- Observation or evidence
26	- Single, specific change
27	- Directional expectation
28	- Defined audience
29	- Measurable success criteria
30
31	---
32
33	### 3️⃣ Hypothesis Lock (Hard Gate)
34
35	Before designing variants or metrics, you MUST:
36
37	- Present the final hypothesis
38	- Specify:
39	- Target audience
40	- Primary metric
41	- Expected direction of effect
42	- Minimum Detectable Effect (MDE)
43
44	Ask explicitly:
45
46	> “Is this the final hypothesis we are committing to for this test?”
47
48	Do NOT proceed until confirmed.
49
50	---
51
52	### 4️⃣ Assumptions & Validity Check (Mandatory)
53
54	Explicitly list assumptions about:
55
56	- Traffic stability
57	- User independence
58	- Metric reliability
59	- Randomization quality
60	- External factors (seasonality, campaigns, releases)
61
62	If assumptions are weak or violated:
63
64	- Warn the user
65	- Recommend delaying or redesigning the test
66
67	---
68
69	### 5️⃣ Test Type Selection
70
71	Choose the simplest valid test:
72
73	- A/B Test – single change, two variants
74	- A/B/n Test – multiple variants, higher traffic required
75	- Multivariate Test (MVT) – interaction effects, very high traffic
76	- Split URL Test – major structural changes
77
78	Default to A/B unless there is a clear reason otherwise.
79
80	---
81
82	### 6️⃣ Metrics Definition
83
84	#### Primary Metric (Mandatory)
85
86	- Single metric used to evaluate success
87	- Directly tied to the hypothesis
88	- Pre-defined and frozen before launch
89
90	#### Secondary Metrics
91
92	- Provide context
93	- Explain _why_ results occurred
94	- Must not override the primary metric
95
96	#### Guardrail Metrics
97
98	- Metrics that must not degrade
99	- Used to prevent harmful wins
100	- Trigger test stop if significantly negative
101
102	---
103
104	### 7️⃣ Sample Size & Duration
105
106	Define upfront:
107
108	- Baseline rate
109	- MDE
110	- Significance level (typically 95%)
111	- Statistical power (typically 80%)
112
113	Estimate:
114
115	- Required sample size per variant
116	- Expected test duration
117
118	Do NOT proceed without a realistic sample size estimate.
119
120	---
121
122	### 8️⃣ Execution Readiness Gate (Hard Stop)
123
124	You may proceed to implementation only if all are true:
125
126	- Hypothesis is locked
127	- Primary metric is frozen
128	- Sample size is calculated
129	- Test duration is defined
130	- Guardrails are set
131	- Tracking is verified
132
133	If any item is missing, stop and resolve it.
134
135	---
136
137	## Running the Test
138
139	### During the Test
140
141	DO:
142
143	- Monitor technical health
144	- Document external factors
145
146	DO NOT:
147
148	- Stop early due to “good-looking” results
149	- Change variants mid-test
150	- Add new traffic sources
151	- Redefine success criteria
152
153	---
154
155	## Analyzing Results
156
157	### Analysis Discipline
158
159	When interpreting results:
160
161	- Do NOT generalize beyond the tested population
162	- Do NOT claim causality beyond the tested change
163	- Do NOT override guardrail failures
164	- Separate statistical significance from business judgment
165
166	### Interpretation Outcomes
167
168	\| Result \| Action \|
169	\| -------------------- \| -------------------------------------- \|
170	\| Significant positive \| Consider rollout \|
171	\| Significant negative \| Reject variant, document learning \|
172	\| Inconclusive \| Consider more traffic or bolder change \|
173	\| Guardrail failure \| Do not ship, even if primary wins \|
174
175	---
176
177	## Documentation & Learning
178
179	### Test Record (Mandatory)
180
181	Document:
182
183	- Hypothesis
184	- Variants
185	- Metrics
186	- Sample size vs achieved
187	- Results
188	- Decision
189	- Learnings
190	- Follow-up ideas
191
192	Store records in a shared, searchable location to avoid repeated failures.
193
194	---
195
196	## Refusal Conditions (Safety)
197
198	Refuse to proceed if:
199
200	- Baseline rate is unknown and cannot be estimated
201	- Traffic is insufficient to detect the MDE
202	- Primary metric is undefined
203	- Multiple variables are changed without proper design
204	- Hypothesis cannot be clearly stated
205
206	Explain why and recommend next steps.
207
208	---
209
210	## Key Principles (Non-Negotiable)
211
212	- One hypothesis per test
213	- One primary metric
214	- Commit before launch
215	- No peeking
216	- Learning over winning
217	- Statistical rigor first
218
219	---
220
221	## Final Reminder
222
223	A/B testing is not about proving ideas right.
224	It is about learning the truth with confidence.
225
226	If you feel tempted to rush, simplify, or “just try it” —
227	that is the signal to slow down and re-check the design.

People who liked this also viewed

6 picks

Type