built-in-metrics

Name: built-in-metrics
Author: launchdarkly

20stars6forks

How to install built-in-metrics?

launchdarkly/ai-tooling/built-in-metrics

$npx -y skills add launchdarkly/ai-tooling --skill built-in-metrics

Installs into the current project.

Summary

Instrument an existing codebase with LaunchDarkly config tracking. Walks the four-tier ladder (managed runner → provider package → custom extractor + trackMetricsOf → raw manual) and picks the lowest-ceremony option that still captures duration, tokens, and success/error.

Files · 1

View on GitHub

SKILL.md

1	# Agent Metrics Instrumentation
2
3	You're using a skill that wires LaunchDarkly agent metrics around an existing provider call. Your job is to audit what's already there, pick the right tier from the ladder below, and implement it with the least ceremony that still captures the metrics the Monitoring tab needs (duration, input/output tokens, success/error, plus TTFT when streaming).
4
5	The single most important thing to get right: default to the highest tier that fits the shape of the call. Going lower ("just write the manual tracker calls") looks flexible but costs you drift, missed metrics, and legacy patterns the SDKs have moved past.
6
7	## The four-tier ladder
8
9	This is the order the official SDK READMEs (Python core, Node core, and every provider package) recommend. Walk from the top and stop at the first tier that fits:
10
11	\| Tier \| Pattern \| Use when \| Tracks automatically \|
12	\|------\|---------\|----------\|----------------------\|
13	\| 1 — Managed runner \| Python: `ai_client.create_model(...)` returning a `ManagedModel`, then `await model.run(...)`. <br>Node: `aiClient.createModel(...)` returning a `ManagedModel`, then `await model.run(...)`. \| The call is conversational (chat history, turn-based). This is what the provider READMEs lead with. \| Duration, tokens, success/error — all of it, zero tracker calls. \|
14	\| 2 — Provider package + `trackMetricsOf` \| `tracker.trackMetricsOf(Provider.getAIMetricsFromResponse, () => providerCall())`. Provider packages today: `@launchdarkly/server-sdk-ai-openai`, `-langchain`, `-vercel` (Node) and `launchdarkly-server-sdk-ai-openai`, `-langchain` (Python). \| The shape isn't a chat loop (one-shot completion, structured output, agent step) but the framework or provider has a package. \| Duration + success/error from the wrapper; tokens from the package's built-in `getAIMetricsFromResponse` extractor. \|
15	\| 3 — Custom extractor + `trackMetricsOf` \| Same `trackMetricsOf` wrapper, but you write a small function that maps the provider response to `LDAIMetrics` (tokens + success). \| No provider package exists (Anthropic direct, Gemini, Cohere, custom HTTP). \| Duration + success/error from the wrapper; tokens from your extractor. \|
16	\| 4 — Raw manual \| Separate calls to `trackDuration`, `trackTokens`, `trackSuccess` / `trackError`, plus `trackTimeToFirstToken` for streams. \| Streaming with TTFT, unusual response shapes, partial tracking, anything Tier 2–3 can't cleanly wrap. \| Only what you explicitly call — it's on you to not miss one. \|
17
18	Every provider — OpenAI, LangChain, Vercel, Bedrock, Anthropic, Gemini, custom HTTP — uses the same generic shape: `tracker.trackMetricsOf(getAIMetricsFromResponse, () => providerCall())` in Node, `tracker.track_metrics_of(get_ai_metrics_from_response, provider_call)` in Python. The extractor is the only thing that changes per provider: import `getAIMetricsFromResponse` from the matching `@launchdarkly/server-sdk-ai-<provider>` (or `ldai_<provider>`) package, or write a small custom function that returns `LDAIMetrics`. There are no provider-specific tracker methods.
19
20	## Workflow
21
22	### 1. Explore the existing call site
23
24	Before picking a tier, find the provider call and answer these questions:
25
26	- [ ] Shape? Is it a chat loop (history + turn-based), a one-shot completion, an agent step, or something else? → drives Tier 1 vs 2.
27	- [ ] Framework? Raw provider SDK? LangChain / LangGraph? Vercel AI SDK? CrewAI? Strands? → drives which Tier-2 provider package (if any) applies.
28	- [ ] Provider? OpenAI, Anthropic, Bedrock, Gemini, Azure, custom HTTP? → cross-reference with the package availability matrix below.
29	- [ ] Streaming? If yes, you'll need TTFT tracking, which means Tier 4 for the TTFT part even if the rest is Tier 2.
30	- [ ] Language? Python or Node? Provider-package coverage differs between them.
31	- [ ] Already using a config? If not, route to `configs-create` first — tracking requires a tracker, which is obtained by calling `create_tracker()` / `createTracker()` on the config object returned by `completion_config()` / `completionConfig()` / `createModel()`.
32	- [ ] On the current SDK API? If the call site uses `aiclient.config(...)` / `aiClient.config(...)` or constructs an `AIConfig(...)` / `LDAIConfig` default, it's on the pre-0.20 surface. Migrate it as part of this work before adding tracking:
33	- `aiclient.config(...)` → `aiclien

People who liked this also viewed

6 picks

Type

built-in-metrics

20stars6forks

View on GitHub

How to install built-in-metrics?

launchdarkly/ai-tooling/built-in-metrics

$npx -y skills add launchdarkly/ai-tooling --skill built-in-metrics

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Agent Metrics Instrumentation
2
3	You're using a skill that wires LaunchDarkly agent metrics around an existing provider call. Your job is to audit what's already there, pick the right tier from the ladder below, and implement it with the least ceremony that still captures the metrics the Monitoring tab needs (duration, input/output tokens, success/error, plus TTFT when streaming).
4
5	The single most important thing to get right: default to the highest tier that fits the shape of the call. Going lower ("just write the manual tracker calls") looks flexible but costs you drift, missed metrics, and legacy patterns the SDKs have moved past.
6
7	## The four-tier ladder
8
9	This is the order the official SDK READMEs (Python core, Node core, and every provider package) recommend. Walk from the top and stop at the first tier that fits:
10
11	\| Tier \| Pattern \| Use when \| Tracks automatically \|
12	\|------\|---------\|----------\|----------------------\|
13	\| 1 — Managed runner \| Python: `ai_client.create_model(...)` returning a `ManagedModel`, then `await model.run(...)`. <br>Node: `aiClient.createModel(...)` returning a `ManagedModel`, then `await model.run(...)`. \| The call is conversational (chat history, turn-based). This is what the provider READMEs lead with. \| Duration, tokens, success/error — all of it, zero tracker calls. \|
14	\| 2 — Provider package + `trackMetricsOf` \| `tracker.trackMetricsOf(Provider.getAIMetricsFromResponse, () => providerCall())`. Provider packages today: `@launchdarkly/server-sdk-ai-openai`, `-langchain`, `-vercel` (Node) and `launchdarkly-server-sdk-ai-openai`, `-langchain` (Python). \| The shape isn't a chat loop (one-shot completion, structured output, agent step) but the framework or provider has a package. \| Duration + success/error from the wrapper; tokens from the package's built-in `getAIMetricsFromResponse` extractor. \|
15	\| 3 — Custom extractor + `trackMetricsOf` \| Same `trackMetricsOf` wrapper, but you write a small function that maps the provider response to `LDAIMetrics` (tokens + success). \| No provider package exists (Anthropic direct, Gemini, Cohere, custom HTTP). \| Duration + success/error from the wrapper; tokens from your extractor. \|
16	\| 4 — Raw manual \| Separate calls to `trackDuration`, `trackTokens`, `trackSuccess` / `trackError`, plus `trackTimeToFirstToken` for streams. \| Streaming with TTFT, unusual response shapes, partial tracking, anything Tier 2–3 can't cleanly wrap. \| Only what you explicitly call — it's on you to not miss one. \|
17
18	Every provider — OpenAI, LangChain, Vercel, Bedrock, Anthropic, Gemini, custom HTTP — uses the same generic shape: `tracker.trackMetricsOf(getAIMetricsFromResponse, () => providerCall())` in Node, `tracker.track_metrics_of(get_ai_metrics_from_response, provider_call)` in Python. The extractor is the only thing that changes per provider: import `getAIMetricsFromResponse` from the matching `@launchdarkly/server-sdk-ai-<provider>` (or `ldai_<provider>`) package, or write a small custom function that returns `LDAIMetrics`. There are no provider-specific tracker methods.
19
20	## Workflow
21
22	### 1. Explore the existing call site
23
24	Before picking a tier, find the provider call and answer these questions:
25
26	- [ ] Shape? Is it a chat loop (history + turn-based), a one-shot completion, an agent step, or something else? → drives Tier 1 vs 2.
27	- [ ] Framework? Raw provider SDK? LangChain / LangGraph? Vercel AI SDK? CrewAI? Strands? → drives which Tier-2 provider package (if any) applies.
28	- [ ] Provider? OpenAI, Anthropic, Bedrock, Gemini, Azure, custom HTTP? → cross-reference with the package availability matrix below.
29	- [ ] Streaming? If yes, you'll need TTFT tracking, which means Tier 4 for the TTFT part even if the rest is Tier 2.
30	- [ ] Language? Python or Node? Provider-package coverage differs between them.
31	- [ ] Already using a config? If not, route to `configs-create` first — tracking requires a tracker, which is obtained by calling `create_tracker()` / `createTracker()` on the config object returned by `completion_config()` / `completionConfig()` / `createModel()`.
32	- [ ] On the current SDK API? If the call site uses `aiclient.config(...)` / `aiClient.config(...)` or constructs an `AIConfig(...)` / `LDAIConfig` default, it's on the pre-0.20 surface. Migrate it as part of this work before adding tracking:
33	- `aiclient.config(...)` → `aiclien

People who liked this also viewed

6 picks

Type