arize-prompt-optimization

Name: arize-prompt-optimization
Author: github

37kstars4.6kforks

How to install arize-prompt-optimization?

github/awesome-copilot/arize-prompt-optimization

$npx -y skills add github/awesome-copilot --skill arize-prompt-optimization

Installs into the current project.

Summary

Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, m

Files · 1

View on GitHub

SKILL.md

1	# Arize Prompt Optimization Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	## Concepts
6
7	### Where Prompts Live in Trace Data
8
9	LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:
10
11	\| Column \| What it contains \| When to use \|
12	\|--------\|-----------------\|-------------\|
13	\| `attributes.llm.input_messages` \| Structured chat messages (system, user, assistant, tool) in role-based format \| Primary source for chat-based LLM prompts \|
14	\| `attributes.llm.input_messages.roles` \| Array of roles: `system`, `user`, `assistant`, `tool` \| Extract individual message roles \|
15	\| `attributes.llm.input_messages.contents` \| Array of message content strings \| Extract message text \|
16	\| `attributes.input.value` \| Serialized prompt or user question (generic, all span kinds) \| Fallback when structured messages are not available \|
17	\| `attributes.llm.prompt_template.template` \| Template with `{variable}` placeholders (e.g., `"Answer {question} using {context}"`) \| When the app uses prompt templates \|
18	\| `attributes.llm.prompt_template.variables` \| Template variable values (JSON object) \| See what values were substituted into the template \|
19	\| `attributes.output.value` \| Model response text \| See what the LLM produced \|
20	\| `attributes.llm.output_messages` \| Structured model output (including tool calls) \| Inspect tool-calling responses \|
21
22	### Finding Prompts by Span Kind
23
24	- LLM span (`attributes.openinference.span.kind = 'LLM'`): Check `attributes.llm.input_messages` for structured chat messages, OR `attributes.input.value` for a serialized prompt. Check `attributes.llm.prompt_template.template` for the template.
25	- Chain/Agent span: `attributes.input.value` contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.
26	- Tool span: `attributes.input.value` has tool input, `attributes.output.value` has tool result. Not typically where prompts live.
27
28	### Performance Signal Columns
29
30	These columns carry the feedback data used for optimization:
31
32	\| Column pattern \| Source \| What it tells you \|
33	\|---------------\|--------\|-------------------\|
34	\| `annotation.<name>.label` \| Human reviewers \| Categorical grade (e.g., `correct`, `incorrect`, `partial`) \|
35	\| `annotation.<name>.score` \| Human reviewers \| Numeric quality score (e.g., 0.0 - 1.0) \|
36	\| `annotation.<name>.text` \| Human reviewers \| Freeform explanation of the grade \|
37	\| `eval.<name>.label` \| LLM-as-judge evals \| Automated categorical assessment \|
38	\| `eval.<name>.score` \| LLM-as-judge evals \| Automated numeric score \|
39	\| `eval.<name>.explanation` \| LLM-as-judge evals \| Why the eval gave that score -- most valuable for optimization \|
40	\| `attributes.input.value` \| Trace data \| What went into the LLM \|
41	\| `attributes.output.value` \| Trace data \| What the LLM produced \|
42	\| `{experiment_name}.output` \| Experiment runs \| Output from a specific experiment \|
43
44	## Prerequisites
45
46	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
47
48	If an `ax` command fails, troubleshoot based on the error:
49	- `command not found` or version error → see references/ax-setup.md
50	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
51	- Space unknown → run `ax spaces list` to pick by name, or ask the user
52	- Project unclear → ask the user, or run `ax projects list -o json --limit 100` and present as selectable options
53	- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
54	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
55
56	## Phase 1: Extract the Current Promp

People who liked this also viewed

6 picks

Type

arize-prompt-optimization

37kstars4.6kforks

View on GitHub

How to install arize-prompt-optimization?

github/awesome-copilot/arize-prompt-optimization

$npx -y skills add github/awesome-copilot --skill arize-prompt-optimization

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Arize Prompt Optimization Skill
2
3	> `SPACE` — All `--space` flags and the `ARIZE_SPACE` env var accept a space name (e.g., `my-workspace`) or a base64 space ID (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.
4
5	## Concepts
6
7	### Where Prompts Live in Trace Data
8
9	LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:
10
11	\| Column \| What it contains \| When to use \|
12	\|--------\|-----------------\|-------------\|
13	\| `attributes.llm.input_messages` \| Structured chat messages (system, user, assistant, tool) in role-based format \| Primary source for chat-based LLM prompts \|
14	\| `attributes.llm.input_messages.roles` \| Array of roles: `system`, `user`, `assistant`, `tool` \| Extract individual message roles \|
15	\| `attributes.llm.input_messages.contents` \| Array of message content strings \| Extract message text \|
16	\| `attributes.input.value` \| Serialized prompt or user question (generic, all span kinds) \| Fallback when structured messages are not available \|
17	\| `attributes.llm.prompt_template.template` \| Template with `{variable}` placeholders (e.g., `"Answer {question} using {context}"`) \| When the app uses prompt templates \|
18	\| `attributes.llm.prompt_template.variables` \| Template variable values (JSON object) \| See what values were substituted into the template \|
19	\| `attributes.output.value` \| Model response text \| See what the LLM produced \|
20	\| `attributes.llm.output_messages` \| Structured model output (including tool calls) \| Inspect tool-calling responses \|
21
22	### Finding Prompts by Span Kind
23
24	- LLM span (`attributes.openinference.span.kind = 'LLM'`): Check `attributes.llm.input_messages` for structured chat messages, OR `attributes.input.value` for a serialized prompt. Check `attributes.llm.prompt_template.template` for the template.
25	- Chain/Agent span: `attributes.input.value` contains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree.
26	- Tool span: `attributes.input.value` has tool input, `attributes.output.value` has tool result. Not typically where prompts live.
27
28	### Performance Signal Columns
29
30	These columns carry the feedback data used for optimization:
31
32	\| Column pattern \| Source \| What it tells you \|
33	\|---------------\|--------\|-------------------\|
34	\| `annotation.<name>.label` \| Human reviewers \| Categorical grade (e.g., `correct`, `incorrect`, `partial`) \|
35	\| `annotation.<name>.score` \| Human reviewers \| Numeric quality score (e.g., 0.0 - 1.0) \|
36	\| `annotation.<name>.text` \| Human reviewers \| Freeform explanation of the grade \|
37	\| `eval.<name>.label` \| LLM-as-judge evals \| Automated categorical assessment \|
38	\| `eval.<name>.score` \| LLM-as-judge evals \| Automated numeric score \|
39	\| `eval.<name>.explanation` \| LLM-as-judge evals \| Why the eval gave that score -- most valuable for optimization \|
40	\| `attributes.input.value` \| Trace data \| What went into the LLM \|
41	\| `attributes.output.value` \| Trace data \| What the LLM produced \|
42	\| `{experiment_name}.output` \| Experiment runs \| Output from a specific experiment \|
43
44	## Prerequisites
45
46	Proceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.
47
48	If an `ax` command fails, troubleshoot based on the error:
49	- `command not found` or version error → see references/ax-setup.md
50	- `401 Unauthorized` / missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys
51	- Space unknown → run `ax spaces list` to pick by name, or ask the user
52	- Project unclear → ask the user, or run `ax projects list -o json --limit 100` and present as selectable options
53	- LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill
54	- Security: Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. If credentials are not available through these channels, ask the user.
55
56	## Phase 1: Extract the Current Promp

People who liked this also viewed

6 picks

Type