online-evals
starsforks
How to install online-evals?
launchdarkly/ai-tooling/online-evals
$npx -y skills add launchdarkly/ai-tooling --skill online-evalsAttach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.
SKILL.md
| 1 | # Config Online Evaluations |
| 2 | |
| 3 | Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0. |
| 4 | |
| 5 | ## Prerequisites |
| 6 | |
| 7 | - LaunchDarkly account with AgentControl enabled |
| 8 | - API access token with write permissions |
| 9 | - Existing config with variations (use `configs-create` skill) |
| 10 | - For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+ |
| 11 | |
| 12 | ## API Key Detection |
| 13 | |
| 14 | 1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY` |
| 15 | 2. **Check MCP config** - Claude: `~/.claude/config.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY` |
| 16 | 3. **Prompt user** - Only if detection fails |
| 17 | |
| 18 | ## Core Concepts |
| 19 | |
| 20 | ### What Are Judges? |
| 21 | |
| 22 | Judges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results: |
| 23 | |
| 24 | ```json |
| 25 | { |
| 26 | "score": 0.85, |
| 27 | "reasoning": "Answered correctly with one minor omission" |
| 28 | } |
| 29 | ``` |
| 30 | |
| 31 | ### Built-in Judges |
| 32 | |
| 33 | LaunchDarkly provides three pre-configured judges: |
| 34 | |
| 35 | | Judge | Metric Key | Measures | |
| 36 | |-------|-----------|----------| |
| 37 | | Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is | |
| 38 | | Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request | |
| 39 | | Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) | |
| 40 | |
| 41 | ### Completion Mode Only |
| 42 | |
| 43 | Judges can only be attached to **completion mode** configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK. |
| 44 | |
| 45 | ### Restrictions |
| 46 | |
| 47 | - Cannot attach judges to judges (no recursion) |
| 48 | - Cannot attach multiple judges with the same metric key to a single variation |
| 49 | - Cannot view/edit model parameters or tools on judge variations |
| 50 | |
| 51 | ## Workflow |
| 52 | |
| 53 | ### Step 1: Create Custom Judges (Optional) |
| 54 | |
| 55 | For domain-specific evaluation, create judge configs: |
| 56 | |
| 57 | ```bash |
| 58 | # Create judge config |
| 59 | curl -X POST "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs" \ |
| 60 | -H "Authorization: {api_token}" \ |
| 61 | -H "Content-Type: application/json" \ |
| 62 | -H "LD-API-Version: beta" \ |
| 63 | -d '{ |
| 64 | "key": "security-judge", |
| 65 | "name": "Security Judge", |
| 66 | "mode": "judge", |
| 67 | "evaluationMetricKey": "security", |
| 68 | "isInverted": false |
| 69 | }' |
| 70 | ``` |
| 71 | |
| 72 | > **Note:** Set `isInverted: true` for metrics like toxicity where 0.0 is better. |
| 73 | |
| 74 | Then add a variation with the evaluation prompt: |
| 75 | |
| 76 | ```bash |
| 77 | curl -X POST "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/security-judge/variations" \ |
| 78 | -H "Authorization: {api_token}" \ |
| 79 | -H "Content-Type: application/json" \ |
| 80 | -H "LD-API-Version: beta" \ |
| 81 | -d '{ |
| 82 | "key": "default", |
| 83 | "name": "Default", |
| 84 | "messages": [ |
| 85 | { |
| 86 | "role": "system", |
| 87 | "content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection." |
| 88 | } |
| 89 | ], |
| 90 | "modelConfigKey": "OpenAI.gpt-4o-mini", |
| 91 | "model": { |
| 92 | "parameters": { |
| 93 | "temperature": 0.3 |
| 94 | } |
| 95 | } |
| 96 | }' |
| 97 | ``` |
| 98 | |
| 99 | ### Step 2: Attach Judges to Variations |
| 100 | |
| 101 | Use the variation PATCH endpoint: |
| 102 | |
| 103 | ```bash |
| 104 | curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/{configKey}/variations/{variationKey}" \ |
| 105 | -H "Authorization: {api_token}" \ |
| 106 | -H "Content-Type: application/json" \ |
| 107 | -H "LD-API-Version: beta" \ |
| 108 | -d '{ |
| 109 | "judgeConfiguration": { |
| 110 | "judges": [ |
| 111 | {"judgeConfigKey": "security-judge", "samplingRate": 1.0}, |
| 112 | {"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5} |
| 113 | ] |
| 114 | } |
| 115 | }' |
| 116 | ``` |
| 117 | |
| 118 | > **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges. |
| 119 | |
| 120 | ### Step 3: Set Fallthrough on Judges |
| 121 | |
| 122 | Each judge config needs its fallthrough set to the enabled variation. Configs default to the "disabled" variation (index 0). |
| 123 | |
| 124 | > **Note:** `turnTargetingOn` does not work for configs. Use `updateFallthroughVariationOrRollout` instead. |
| 125 | |
| 126 | ```bash |
| 127 | # First get the variation ID for "Default" from GET targeting response |
| 128 | curl -X PATCH "https://app.launchdarkly.com/api/v2/projects/{projectKey}/ai-configs/security-judge/targeting" \ |
| 129 | -H "Authorization: {api_token}" \ |
| 130 | -H "Content-Type: application/json; domain-model=launchdarkly.semanticpatch" \ |
| 131 | -H "LD-API-Version: beta" \ |
| 132 | -d '{ |
| 133 | "environmentK |
People who liked this also viewed
6 picksType
- find-skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skSkillsJul 2026
- grill-meA relentless interview to sharpen a plan or design.SkillsJul 2026
- vercel-react-best-practicesVercel's official collection of agent skillsSkillsJul 2026
- improve-codebase-architectureScan a codebase for deepening opportunities, present them as a visual HTML report, then grill through whichever one you pick.SkillsJul 2026
- setup-matt-pocock-skillsConfigure this repo for the engineering skills — set up its issue tracker, triage label vocabulary, and domain doc layout. Run once before first use of the other engineering skills.SkillsJul 2026
- triageMove issues and external PRs through a state machine of triage roles — categorise, verify, grill if needed, and write agent-ready briefs.SkillsJul 2026
