launchdarkly-experiment-setup

Name: launchdarkly-experiment-setup
Author: launchdarkly

20stars6forks

How to install launchdarkly-experiment-setup?

launchdarkly/ai-tooling/launchdarkly-experiment-setup

$npx -y skills add launchdarkly/ai-tooling --skill launchdarkly-experiment-setup

Installs into the current project.

Summary

Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner.

Files · 1

View on GitHub

SKILL.md

1	# LaunchDarkly Experiment Setup
2
3	You're using a skill that guides you through setting up and running experiments in LaunchDarkly. Your job is to design the experiment, create it with the right metrics, treatments, and flag config, start data collection, evolve the design between iterations when needed, and stop with a winner.
4
5	## Prerequisites
6
7	This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment.
8
9	Required MCP tools:
10	- `create-experiment` — create a new experiment with its initial iteration (hypothesis, metrics, treatments, flag config).
11	- `start-experiment-iteration` — begin collecting data for an experiment's current draft iteration.
12	- `get-experiment` — check experiment status, treatments, metrics, and current iteration.
13
14	Optional MCP tools:
15	- `list-experiments` — browse existing experiments in the project.
16	- `update-experiment` — update fields on the experiment or its current iteration. Honours `mutableFieldsByStatus`, so what's editable depends on whether the iteration is `not_started`, `running`, or `stopped`. Returns rejected inputs under `skipped`.
17	- `save-and-start-experiment-iteration` — the API-recommended way to change locked fields on a running experiment. Stops the current iteration, creates a new draft with the supplied field updates, and starts it in one call.
18	- `stop-experiment-iteration` — stop the running iteration. You must declare a winner: pass the `winningTreatmentId` (and a `winningReason`). If no variation outperformed, pick the baseline/control as the winner.
19	- `list-metrics`, `create-metric`, `list-metric-events` — manage metrics referenced by the experiment.
20
21	## Core Concepts
22
23	### What Are Experiments?
24
25	Experiments in LaunchDarkly measure the impact of feature flag variations on key metrics. An experiment consists of:
26
27	- Treatments: the flag variations being compared (control vs. test). Each treatment has an `allocationPercent`; the values across treatments should sum to 100.
28	- Metrics: what you're measuring (conversion rate, latency, revenue, etc.). One must be the primary metric.
29	- Flag config: the `flagKey`, `ruleId`, and `flagConfigVersion` of the targeting rule that drives the experiment.
30	- Iteration: a single data-collection window. Created in `not_started` status, becomes `running` when started, transitions to `stopped` when ended.
31	- Holdout (optional): a project-level group of users excluded from the experiment for baseline measurement (`holdoutId`).
32
33	### Experiment Lifecycle
34
35	1. Create the experiment with its first iteration (`create-experiment`).
36	2. Start the iteration to begin data collection (`start-experiment-iteration`).
37	3. Monitor results as data accumulates (`get-experiment`).
38	4. Evolve the design mid-experiment if needed — change locked fields like `treatments`, `metrics`, or `methodology` by calling `save-and-start-experiment-iteration`, which stops the current iteration, creates a new draft with your changes, and starts it.
39	5. Stop the iteration when you have a winner or a clear call (`stop-experiment-iteration`).
40	6. Ship the winning variation.
41
42	## Core Principles
43
44	1. Metrics first: ensure the metrics you'll reference exist before creating the experiment.
45	2. Clear hypothesis: every iteration requires a `hypothesis` string; state what you expect to improve and by how much.
46	3. Proper controls: exactly one treatment must have `baseline: true`.
47	4. Sufficient sample size: let iterations run long enough for statistical significance.
48	5. One change at a time: test one variable per experiment for clear attribution.
49
50	## Workflow
51
52	### Step 1: Prepare Metrics
53
54	1. Use `list-metrics` to find existing metrics.
55	2. If you need a new one, use `create-metric` and note the key.
56	3. Decide which is the primary metric (a single metric or a funnel group). You'll pass its key as `primarySingleMetricKey` or `primaryFunnelKey` on the iteration.
57
58	\| Goal \| Metric type \| Example key \|
59	\|------\|-------------\|-------------\|
60	\| Conversion \| Custom conversion \| `checkout-completed` \|
61	\| Performance \| Custom numeric \| `page-load-time-ms` \|
62	\| Engagement \| Custom conversion \| `feature-clicked` \|
63	\| Revenue \| Custom numeric \| `order-value` \|
64
65	### Step 2: Identify the Targeting Rule
66
67	You need the `ruleId` and current `flagConfigVersion` of the flag rule that will drive the experiment. Use `get-flag` on the flag (or its environment-scoped status) to find them. The fallthrough rule's id is the string `"fallthrough"`.
68
69	### Step 3: Create the Experiment
70
71	Call `create-experiment`. The top-level fi

People who liked this also viewed

6 picks

Type

launchdarkly-experiment-setup

20stars6forks

View on GitHub

How to install launchdarkly-experiment-setup?

launchdarkly/ai-tooling/launchdarkly-experiment-setup

$npx -y skills add launchdarkly/ai-tooling --skill launchdarkly-experiment-setup

Installs into the current project.

Summary

Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner.

Files · 1

View on GitHub

SKILL.md

1	# LaunchDarkly Experiment Setup
2
3	You're using a skill that guides you through setting up and running experiments in LaunchDarkly. Your job is to design the experiment, create it with the right metrics, treatments, and flag config, start data collection, evolve the design between iterations when needed, and stop with a winner.
4
5	## Prerequisites
6
7	This skill requires the remotely hosted LaunchDarkly MCP server to be configured in your environment.
8
9	Required MCP tools:
10	- `create-experiment` — create a new experiment with its initial iteration (hypothesis, metrics, treatments, flag config).
11	- `start-experiment-iteration` — begin collecting data for an experiment's current draft iteration.
12	- `get-experiment` — check experiment status, treatments, metrics, and current iteration.
13
14	Optional MCP tools:
15	- `list-experiments` — browse existing experiments in the project.
16	- `update-experiment` — update fields on the experiment or its current iteration. Honours `mutableFieldsByStatus`, so what's editable depends on whether the iteration is `not_started`, `running`, or `stopped`. Returns rejected inputs under `skipped`.
17	- `save-and-start-experiment-iteration` — the API-recommended way to change locked fields on a running experiment. Stops the current iteration, creates a new draft with the supplied field updates, and starts it in one call.
18	- `stop-experiment-iteration` — stop the running iteration. You must declare a winner: pass the `winningTreatmentId` (and a `winningReason`). If no variation outperformed, pick the baseline/control as the winner.
19	- `list-metrics`, `create-metric`, `list-metric-events` — manage metrics referenced by the experiment.
20
21	## Core Concepts
22
23	### What Are Experiments?
24
25	Experiments in LaunchDarkly measure the impact of feature flag variations on key metrics. An experiment consists of:
26
27	- Treatments: the flag variations being compared (control vs. test). Each treatment has an `allocationPercent`; the values across treatments should sum to 100.
28	- Metrics: what you're measuring (conversion rate, latency, revenue, etc.). One must be the primary metric.
29	- Flag config: the `flagKey`, `ruleId`, and `flagConfigVersion` of the targeting rule that drives the experiment.
30	- Iteration: a single data-collection window. Created in `not_started` status, becomes `running` when started, transitions to `stopped` when ended.
31	- Holdout (optional): a project-level group of users excluded from the experiment for baseline measurement (`holdoutId`).
32
33	### Experiment Lifecycle
34
35	1. Create the experiment with its first iteration (`create-experiment`).
36	2. Start the iteration to begin data collection (`start-experiment-iteration`).
37	3. Monitor results as data accumulates (`get-experiment`).
38	4. Evolve the design mid-experiment if needed — change locked fields like `treatments`, `metrics`, or `methodology` by calling `save-and-start-experiment-iteration`, which stops the current iteration, creates a new draft with your changes, and starts it.
39	5. Stop the iteration when you have a winner or a clear call (`stop-experiment-iteration`).
40	6. Ship the winning variation.
41
42	## Core Principles
43
44	1. Metrics first: ensure the metrics you'll reference exist before creating the experiment.
45	2. Clear hypothesis: every iteration requires a `hypothesis` string; state what you expect to improve and by how much.
46	3. Proper controls: exactly one treatment must have `baseline: true`.
47	4. Sufficient sample size: let iterations run long enough for statistical significance.
48	5. One change at a time: test one variable per experiment for clear attribution.
49
50	## Workflow
51
52	### Step 1: Prepare Metrics
53
54	1. Use `list-metrics` to find existing metrics.
55	2. If you need a new one, use `create-metric` and note the key.
56	3. Decide which is the primary metric (a single metric or a funnel group). You'll pass its key as `primarySingleMetricKey` or `primaryFunnelKey` on the iteration.
57
58	\| Goal \| Metric type \| Example key \|
59	\|------\|-------------\|-------------\|
60	\| Conversion \| Custom conversion \| `checkout-completed` \|
61	\| Performance \| Custom numeric \| `page-load-time-ms` \|
62	\| Engagement \| Custom conversion \| `feature-clicked` \|
63	\| Revenue \| Custom numeric \| `order-value` \|
64
65	### Step 2: Identify the Targeting Rule
66
67	You need the `ruleId` and current `flagConfigVersion` of the flag rule that will drive the experiment. Use `get-flag` on the flag (or its environment-scoped status) to find them. The fallthrough rule's id is the string `"fallthrough"`.
68
69	### Step 3: Create the Experiment
70
71	Call `create-experiment`. The top-level fi

People who liked this also viewed

6 picks

Type