arize-evaluator

Name: arize-evaluator
Author: github

37kstars4.6kforks

How to install arize-evaluator?

github/awesome-copilot/arize-evaluator

$npx -y skills add github/awesome-copilot --skill arize-evaluator

Installs into the current project.

Summary

Handles LLM-as-judge evaluation workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, hallu

Files · 1

View on GitHub

SKILL.md

1	# Arize Evaluator Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	This skill covers designing, creating, and running LLM-as-judge evaluators on Arize. An evaluator defines the judge; a task is how you run it against real data.
6
7	---
8
9	## Prerequisites
10
11	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
12
13	If an `ax` command fails, troubleshoot based on the error:
14	- `command not found` or version error → see references/ax-setup.md
15	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
16	- Space unknown → run `ax spaces list` to pick by name, or ask the user
17	- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
18	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
19	- CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with `ax ai-integrations list`, (4) contact support at https://arize.com/support
20
21	---
22
23	## Concepts
24
25	### What is an Evaluator?
26
27	An evaluator is an LLM-as-judge definition. It contains:
28
29	\| Field \| Description \|
30	\|-------\|-------------\|
31	\| Template \| The judge prompt. Uses `{variable}` placeholders (e.g. `{input}`, `{output}`, `{context}`) that get filled in at run time via a task's column mappings. \|
32	\| Classification choices \| The set of allowed output labels (e.g. `factual` / `hallucinated`). Binary is the default and most common. Each choice can optionally carry a numeric score. \|
33	\| AI Integration \| Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. \|
34	\| Model \| The specific judge model (e.g. `gpt-4o`, `claude-sonnet-4-5`). \|
35	\| Invocation params \| Optional JSON of model settings like `{"temperature": 0}`. Low temperature is recommended for reproducibility. \|
36	\| Optimization direction \| Whether higher scores are better (`maximize`) or worse (`minimize`). Sets how the UI renders trends. \|
37	\| Data granularity \| Whether the evaluator runs at the span, trace, or session level. Most evaluators run at the span level. \|
38
39	Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.
40
41	### What is a Task?
42
43	A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:
44
45	\| Field \| Description \|
46	\|-------\|-------------\|
47	\| Evaluators \| List of evaluators to run. You can run multiple in one task. \|
48	\| Column mappings \| Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. `"input" → "attributes.input.value"`). This is what makes evaluators portable across projects and experiments. \|
49	\| Query filter \| SQL-style expression to select which spans/runs to evaluate (e.g. `"span_kind = 'LLM'"`). Optional but important for precision. \|
50	\| Continuous \| For project tasks: whether to automatically score new spans as they arrive. \|
51	\| Sampling rate \| For continuous project tasks: fraction of new spans to evaluate (0–1). \|
52
53	---
54
55	## Data Granularity
56
57	The `--data-granularity` flag controls what unit of data the eva

People who liked this also viewed

6 picks

Type

arize-evaluator

37kstars4.6kforks

View on GitHub

How to install arize-evaluator?

github/awesome-copilot/arize-evaluator

$npx -y skills add github/awesome-copilot --skill arize-evaluator

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Arize Evaluator Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	This skill covers designing, creating, and running LLM-as-judge evaluators on Arize. An evaluator defines the judge; a task is how you run it against real data.
6
7	---
8
9	## Prerequisites
10
11	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
12
13	If an `ax` command fails, troubleshoot based on the error:
14	- `command not found` or version error → see references/ax-setup.md
15	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
16	- Space unknown → run `ax spaces list` to pick by name, or ask the user
17	- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
18	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
19	- CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with `ax ai-integrations list`, (4) contact support at https://arize.com/support
20
21	---
22
23	## Concepts
24
25	### What is an Evaluator?
26
27	An evaluator is an LLM-as-judge definition. It contains:
28
29	\| Field \| Description \|
30	\|-------\|-------------\|
31	\| Template \| The judge prompt. Uses `{variable}` placeholders (e.g. `{input}`, `{output}`, `{context}`) that get filled in at run time via a task's column mappings. \|
32	\| Classification choices \| The set of allowed output labels (e.g. `factual` / `hallucinated`). Binary is the default and most common. Each choice can optionally carry a numeric score. \|
33	\| AI Integration \| Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. \|
34	\| Model \| The specific judge model (e.g. `gpt-4o`, `claude-sonnet-4-5`). \|
35	\| Invocation params \| Optional JSON of model settings like `{"temperature": 0}`. Low temperature is recommended for reproducibility. \|
36	\| Optimization direction \| Whether higher scores are better (`maximize`) or worse (`minimize`). Sets how the UI renders trends. \|
37	\| Data granularity \| Whether the evaluator runs at the span, trace, or session level. Most evaluators run at the span level. \|
38
39	Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.
40
41	### What is a Task?
42
43	A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:
44
45	\| Field \| Description \|
46	\|-------\|-------------\|
47	\| Evaluators \| List of evaluators to run. You can run multiple in one task. \|
48	\| Column mappings \| Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. `"input" → "attributes.input.value"`). This is what makes evaluators portable across projects and experiments. \|
49	\| Query filter \| SQL-style expression to select which spans/runs to evaluate (e.g. `"span_kind = 'LLM'"`). Optional but important for precision. \|
50	\| Continuous \| For project tasks: whether to automatically score new spans as they arrive. \|
51	\| Sampling rate \| For continuous project tasks: fraction of new spans to evaluate (0–1). \|
52
53	---
54
55	## Data Granularity
56
57	The `--data-granularity` flag controls what unit of data the eva

People who liked this also viewed

6 picks

Type