arize-annotation

Name: arize-annotation
Author: github

37kstars4.6kforks

How to install arize-annotation?

github/awesome-copilot/arize-annotation

$npx -y skills add github/awesome-copilot --skill arize-annotation

Installs into the current project.

Summary

Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label s

Files · 1

View on GitHub

SKILL.md

1	# Arize Annotation Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	This skill covers annotation configs (the label schema) and annotation queues (human review workflows), as well as programmatically annotating project spans via the Python SDK.
6
7	Direction: Human labeling in Arize attaches values defined by configs to spans, dataset examples, experiment-related records, and queue items in the product UI. This skill covers: `ax annotation-configs`, `ax annotation-queues`, and bulk span updates with `ArizeClient.spans.update_annotations`.
8
9	---
10
11	## Prerequisites
12
13	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
14
15	If an `ax` command fails, troubleshoot based on the error:
16	- `command not found` or version error → see references/ax-setup.md
17	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
18	- Space unknown → run `ax spaces list` to pick by name, or ask the user
19	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
20
21	---
22
23	## Concepts
24
25	### What is an Annotation Config?
26
27	An annotation config defines the schema for a single type of human feedback label. Before anyone can annotate a span, dataset record, experiment output, or queue item, a config must exist for that label in the space.
28
29	\| Field \| Description \|
30	\|-------\|-------------\|
31	\| Name \| Descriptive identifier (e.g. `Correctness`, `Helpfulness`). Must be unique within the space. \|
32	\| Type \| `categorical` (pick from a list), `continuous` (numeric range), or `freeform` (free text). \|
33	\| Values \| For categorical: array of `{"label": str, "score": number}` pairs. \|
34	\| Min/Max Score \| For continuous: numeric bounds. \|
35	\| Optimization Direction \| Whether higher scores are better (`maximize`) or worse (`minimize`). Used to render trends in the UI. \|
36
37	### Where labels get applied (surfaces)
38
39	\| Surface \| Typical path \|
40	\|---------\|----------------\|
41	\| Project spans \| Python SDK `spans.update_annotations` (below) and/or the Arize UI \|
42	\| Dataset examples \| Arize UI (human labeling flows); configs must exist in the space \|
43	\| Experiment outputs \| Often reviewed alongside datasets or traces in the UI — see arize-experiment, arize-dataset \|
44	\| Annotation queue items \| `ax annotation-queues` CLI (below) and/or the Arize UI; configs must exist \|
45
46	Always ensure the relevant annotation config exists in the space before expecting labels to persist.
47
48	---
49
50	## Basic CRUD: Annotation Configs
51
52	### List
53
54	```bash
55	ax annotation-configs list --space SPACE
56	ax annotation-configs list --space SPACE -o json
57	ax annotation-configs list --space SPACE --limit 20
58	```
59
60	### Create — Categorical
61
62	Categorical configs present a fixed set of labels for reviewers to choose from.
63
64	```bash
65	ax annotation-configs create \
66	--name "Correctness" \
67	--space SPACE \
68	--type categorical \
69	--value correct \
70	--value incorrect \
71	--optimization-direction maximize
72	```
73
74	Common binary label pairs:
75	- `correct` / `incorrect`
76	- `helpful` / `unhelpful`
77	- `safe` / `unsafe`
78	- `relevant` / `irrelevant`
79	- `pass` / `fail`
80
81	### Create — Continuous
82
83	Continuous configs let reviewers enter a numeric score within a defined range.
84
85	```bash
86	ax annotation-configs create \
87	--name "Quality Score" \
88	--space SPACE \
89	--type continuous \
90	--min-score 0 \
91	--max-score 10 \
92	--optimization-direction maximize
93	```
94
95	### Create — Freeform
96
97	Freeform configs collect open-ended text feedback. No additional flags needed beyond name, space, and type.
98
99	```bash
100	ax annotation-configs create \
101	--name "Reviewer Notes" \
102	--space SPACE \
103	--type freeform
104	```
105
106	### Get
107
108	```bash
109	ax annotation-configs get NAME_OR_ID
110	ax annotation-configs get NAME_OR_ID -o json
111	ax annotation-configs get NAME_OR_ID --space SPACE # required when using name instead of ID
112	```
113
114	### Delete
115
116	```bash
117	ax annotation-configs delete NA

githubPublisher · 203 listings

Preview

github/awesome-copilot

# Arize Annotation Skill

> **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`).

This skill covers **annotation configs** (the label schema) and **annotation queues** (human review workflows), as well as programmatically annotating project s

**Direction:** Human labeling in Arize attaches values defined by configs to **spans**, **dataset examples**, **experiment-related records**, and **queue items*

Repogithub/awesome-copilot

LicenseMIT

TypeSkills

CategoryTesting

UpdatedJul 2026

Repo stars37k

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type

arize-annotation

37kstars4.6kforks

View on GitHub

How to install arize-annotation?

github/awesome-copilot/arize-annotation

$npx -y skills add github/awesome-copilot --skill arize-annotation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Arize Annotation Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	This skill covers annotation configs (the label schema) and annotation queues (human review workflows), as well as programmatically annotating project spans via the Python SDK.
6
7	Direction: Human labeling in Arize attaches values defined by configs to spans, dataset examples, experiment-related records, and queue items in the product UI. This skill covers: `ax annotation-configs`, `ax annotation-queues`, and bulk span updates with `ArizeClient.spans.update_annotations`.
8
9	---
10
11	## Prerequisites
12
13	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
14
15	If an `ax` command fails, troubleshoot based on the error:
16	- `command not found` or version error → see references/ax-setup.md
17	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
18	- Space unknown → run `ax spaces list` to pick by name, or ask the user
19	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
20
21	---
22
23	## Concepts
24
25	### What is an Annotation Config?
26
27	An annotation config defines the schema for a single type of human feedback label. Before anyone can annotate a span, dataset record, experiment output, or queue item, a config must exist for that label in the space.
28
29	\| Field \| Description \|
30	\|-------\|-------------\|
31	\| Name \| Descriptive identifier (e.g. `Correctness`, `Helpfulness`). Must be unique within the space. \|
32	\| Type \| `categorical` (pick from a list), `continuous` (numeric range), or `freeform` (free text). \|
33	\| Values \| For categorical: array of `{"label": str, "score": number}` pairs. \|
34	\| Min/Max Score \| For continuous: numeric bounds. \|
35	\| Optimization Direction \| Whether higher scores are better (`maximize`) or worse (`minimize`). Used to render trends in the UI. \|
36
37	### Where labels get applied (surfaces)
38
39	\| Surface \| Typical path \|
40	\|---------\|----------------\|
41	\| Project spans \| Python SDK `spans.update_annotations` (below) and/or the Arize UI \|
42	\| Dataset examples \| Arize UI (human labeling flows); configs must exist in the space \|
43	\| Experiment outputs \| Often reviewed alongside datasets or traces in the UI — see arize-experiment, arize-dataset \|
44	\| Annotation queue items \| `ax annotation-queues` CLI (below) and/or the Arize UI; configs must exist \|
45
46	Always ensure the relevant annotation config exists in the space before expecting labels to persist.
47
48	---
49
50	## Basic CRUD: Annotation Configs
51
52	### List
53
54	```bash
55	ax annotation-configs list --space SPACE
56	ax annotation-configs list --space SPACE -o json
57	ax annotation-configs list --space SPACE --limit 20
58	```
59
60	### Create — Categorical
61
62	Categorical configs present a fixed set of labels for reviewers to choose from.
63
64	```bash
65	ax annotation-configs create \
66	--name "Correctness" \
67	--space SPACE \
68	--type categorical \
69	--value correct \
70	--value incorrect \
71	--optimization-direction maximize
72	```
73
74	Common binary label pairs:
75	- `correct` / `incorrect`
76	- `helpful` / `unhelpful`
77	- `safe` / `unsafe`
78	- `relevant` / `irrelevant`
79	- `pass` / `fail`
80
81	### Create — Continuous
82
83	Continuous configs let reviewers enter a numeric score within a defined range.
84
85	```bash
86	ax annotation-configs create \
87	--name "Quality Score" \
88	--space SPACE \
89	--type continuous \
90	--min-score 0 \
91	--max-score 10 \
92	--optimization-direction maximize
93	```
94
95	### Create — Freeform
96
97	Freeform configs collect open-ended text feedback. No additional flags needed beyond name, space, and type.
98
99	```bash
100	ax annotation-configs create \
101	--name "Reviewer Notes" \
102	--space SPACE \
103	--type freeform
104	```
105
106	### Get
107
108	```bash
109	ax annotation-configs get NAME_OR_ID
110	ax annotation-configs get NAME_OR_ID -o json
111	ax annotation-configs get NAME_OR_ID --space SPACE # required when using name instead of ID
112	```
113
114	### Delete
115
116	```bash
117	ax annotation-configs delete NA

githubPublisher · 203 listings

Preview

github/awesome-copilot

# Arize Annotation Skill

> **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`).

This skill covers **annotation configs** (the label schema) and **annotation queues** (human review workflows), as well as programmatically annotating project s

**Direction:** Human labeling in Arize attaches values defined by configs to **spans**, **dataset examples**, **experiment-related records**, and **queue items*

Repogithub/awesome-copilot

LicenseMIT

TypeSkills

CategoryTesting

UpdatedJul 2026

Repo stars37k

First seenJul 22, 2026

People who liked this also viewed

6 picks

Type