inngest-agent-evals

Name: inngest-agent-evals
Author: inngest

50installs5forks

How to install inngest-agent-evals?

inngest/inngest-skills/inngest-agent-evals

$npx -y skills add inngest/inngest-skills --skill inngest-agent-evals

Installs into the current project.

Summary

Use when building, migrating, or debugging Agent Evals on Inngest: scoring AI agent or workflow outcomes, deferred scorers, sessions, traces, step experiments, experiment variant attribution, Insights queries, or production eval loops for prompts, models, tools, providers, and ag

Files · 1

View on GitHub

SKILL.md

1	# Inngest Agent Evals
2
3	Use this skill when the user wants to evaluate AI agents or AI workflows in
4	production, add scoring, compare prompts/models/tools, group related runs, or
5	debug why an agent outcome was good or bad.
6
7	Agent Evals is not a separate package. It is the production evaluation workflow
8	built from Inngest functions, durable steps, scores, deferred scorers, sessions,
9	traces, experiments, and Insights.
10
11	## Canonical References
12
13	Check the public docs first when exact API details matter:
14
15	- Agent Evals overview: https://www.inngest.com/docs/learn/agent-evals
16	- Scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/scoring
17	- Deferred scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/deferred-scoring
18	- Step experiments guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/step-experiments
19	- Sessions: https://www.inngest.com/docs/features/events-triggers/sessions
20	- Traces: https://www.inngest.com/docs/platform/monitor/traces
21	- Insights: https://www.inngest.com/docs/platform/monitor/insights
22	- Scoring reference: https://www.inngest.com/docs/reference/typescript/v4/functions/scoring
23	- `group.experiment()` reference: https://www.inngest.com/docs/reference/typescript/v4/functions/group-experiment
24	- Launch context: https://www.inngest.com/blog/introducing-agent-evals
25
26	## Decision Flow
27
28	Start from the outcome, not the mechanism:
29
30	1. Identify the product or quality signal: helpful click, ticket resolution,
31	conversion, guardrail pass, retrieval quality, model confidence, latency,
32	cost, human review, or LLM-as-judge result.
33	2. Decide when the signal appears:
34	- During the run: use direct scoring with `step.score()` or
35	`inngest.score()`.
36	- After the run: use deferred scoring with `createScorer()` and `defer()`,
37	or explicit `inngest.score({ runId })`.
38	3. Decide how humans will inspect related work:
39	- Use `meta.sessions` for repeated high-cardinality identifiers such as
40	`conversation_id`, `ticket_id`, `agent_run_id`, or `import_id`.
41	- Use Insights for ad hoc historical analysis, low-cardinality filters, SQL
42	aggregates, and broad investigation.
43	4. Decide whether to compare variants:
44	- Use `group.experiment()` for prompt, model, provider, tool, workflow, or
45	rollout comparisons.
46	- Pass `experimentRef` to deferred or later scores when the score should be
47	credited to the selected variant.
48	5. Preserve traceability:
49	- Keep model calls, tool calls, waits, database writes, and scoring work in
50	durable steps so traces explain why the score happened.
51
52	## Setup Requirements
53
54	- Use TypeScript SDK v4 and install the latest SDK: `npm install inngest@latest`.
55	- Scoring and deferred scoring are beta APIs; verify current imports against
56	the docs before shipping user-facing examples.
57	- `step.score()` requires `scoreMiddleware()` from `inngest/experimental` on
58	the Inngest client.
59	- `createScorer()` is imported from `inngest/experimental` and the scorer must
60	be registered in the serve handler with other functions.
61	- Sessions sent through the TypeScript SDK require v4.7.0 or later.
62	- Step experiments require v4.8.0 or later for `group.experiment()` and the
63	`experiment` helper from `inngest`.
64
65	Client shape:
66
67	```typescript
68	import { Inngest } from "inngest";
69	import { scoreMiddleware } from "inngest/experimental";
70
71	export const inngest = new Inngest({
72	id: "support-agent",
73	middleware: [scoreMiddleware()],
74	});
75	```
76
77	## Sessions
78
79	Add sessions when events belong to a user flow, conversation, ticket, import,
80	or agent task that someone will inspect repeatedly.
81
82	```typescript
83	await inngest.send({
84	name: "support/ticket.created",
85	data: {
86	ticketId: "tk_123",
87	message: "I can't sign in.",
88	},
89	meta: {
90	sessions: {
91	ticket_id: "tk_123",
92	},
93	},
94	});
95	```
96
97	Rules:
98
99	- Use stable, non-secret, high-cardinality IDs.
100	- Keep the key generic, such as `conversation_id`; put the actual ID in the
101	value.
102	- Pass sessions explicitly through `step.invoke()` and `step.sendEvent()` when
103	downstream runs should join the same session.
104	- Do not use sessions for labels like `environment: prod`; use Insights for
105	that style of filtering.
106
107	## Direct Scoring
108
109	Use direct scoring when the score is known during the function run.
110
111	Good direct scores:
112
113	- guardrail pass/fail
114	- JSON validity
115	- retrieval confidenc

People who liked this also viewed

6 picks

Type

inngest-agent-evals

50installs5forks

View on GitHub

How to install inngest-agent-evals?

inngest/inngest-skills/inngest-agent-evals

$npx -y skills add inngest/inngest-skills --skill inngest-agent-evals

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Inngest Agent Evals
2
3	Use this skill when the user wants to evaluate AI agents or AI workflows in
4	production, add scoring, compare prompts/models/tools, group related runs, or
5	debug why an agent outcome was good or bad.
6
7	Agent Evals is not a separate package. It is the production evaluation workflow
8	built from Inngest functions, durable steps, scores, deferred scorers, sessions,
9	traces, experiments, and Insights.
10
11	## Canonical References
12
13	Check the public docs first when exact API details matter:
14
15	- Agent Evals overview: https://www.inngest.com/docs/learn/agent-evals
16	- Scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/scoring
17	- Deferred scoring guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/deferred-scoring
18	- Step experiments guide: https://www.inngest.com/docs/features/inngest-functions/steps-workflows/step-experiments
19	- Sessions: https://www.inngest.com/docs/features/events-triggers/sessions
20	- Traces: https://www.inngest.com/docs/platform/monitor/traces
21	- Insights: https://www.inngest.com/docs/platform/monitor/insights
22	- Scoring reference: https://www.inngest.com/docs/reference/typescript/v4/functions/scoring
23	- `group.experiment()` reference: https://www.inngest.com/docs/reference/typescript/v4/functions/group-experiment
24	- Launch context: https://www.inngest.com/blog/introducing-agent-evals
25
26	## Decision Flow
27
28	Start from the outcome, not the mechanism:
29
30	1. Identify the product or quality signal: helpful click, ticket resolution,
31	conversion, guardrail pass, retrieval quality, model confidence, latency,
32	cost, human review, or LLM-as-judge result.
33	2. Decide when the signal appears:
34	- During the run: use direct scoring with `step.score()` or
35	`inngest.score()`.
36	- After the run: use deferred scoring with `createScorer()` and `defer()`,
37	or explicit `inngest.score({ runId })`.
38	3. Decide how humans will inspect related work:
39	- Use `meta.sessions` for repeated high-cardinality identifiers such as
40	`conversation_id`, `ticket_id`, `agent_run_id`, or `import_id`.
41	- Use Insights for ad hoc historical analysis, low-cardinality filters, SQL
42	aggregates, and broad investigation.
43	4. Decide whether to compare variants:
44	- Use `group.experiment()` for prompt, model, provider, tool, workflow, or
45	rollout comparisons.
46	- Pass `experimentRef` to deferred or later scores when the score should be
47	credited to the selected variant.
48	5. Preserve traceability:
49	- Keep model calls, tool calls, waits, database writes, and scoring work in
50	durable steps so traces explain why the score happened.
51
52	## Setup Requirements
53
54	- Use TypeScript SDK v4 and install the latest SDK: `npm install inngest@latest`.
55	- Scoring and deferred scoring are beta APIs; verify current imports against
56	the docs before shipping user-facing examples.
57	- `step.score()` requires `scoreMiddleware()` from `inngest/experimental` on
58	the Inngest client.
59	- `createScorer()` is imported from `inngest/experimental` and the scorer must
60	be registered in the serve handler with other functions.
61	- Sessions sent through the TypeScript SDK require v4.7.0 or later.
62	- Step experiments require v4.8.0 or later for `group.experiment()` and the
63	`experiment` helper from `inngest`.
64
65	Client shape:
66
67	```typescript
68	import { Inngest } from "inngest";
69	import { scoreMiddleware } from "inngest/experimental";
70
71	export const inngest = new Inngest({
72	id: "support-agent",
73	middleware: [scoreMiddleware()],
74	});
75	```
76
77	## Sessions
78
79	Add sessions when events belong to a user flow, conversation, ticket, import,
80	or agent task that someone will inspect repeatedly.
81
82	```typescript
83	await inngest.send({
84	name: "support/ticket.created",
85	data: {
86	ticketId: "tk_123",
87	message: "I can't sign in.",
88	},
89	meta: {
90	sessions: {
91	ticket_id: "tk_123",
92	},
93	},
94	});
95	```
96
97	Rules:
98
99	- Use stable, non-secret, high-cardinality IDs.
100	- Keep the key generic, such as `conversation_id`; put the actual ID in the
101	value.
102	- Pass sessions explicitly through `step.invoke()` and `step.sendEvent()` when
103	downstream runs should join the same session.
104	- Do not use sessions for labels like `environment: prod`; use Insights for
105	that style of filtering.
106
107	## Direct Scoring
108
109	Use direct scoring when the score is known during the function run.
110
111	Good direct scores:
112
113	- guardrail pass/fail
114	- JSON validity
115	- retrieval confidenc

People who liked this also viewed

6 picks

Type