model-evaluation

Name: model-evaluation
Author: awslabs

843stars132forks

How to install model-evaluation?

awslabs/agent-plugins/model-evaluation

$npx -y skills add awslabs/agent-plugins --skill model-evaluation

Installs into the current project.

Summary

Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.

Files · 1

View on GitHub

SKILL.md

1	# Model Evaluation
2
3	Generate code that evaluates a SageMaker model.
4
5	## Prerequisites
6
7	- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the `sdk-getting-started` skill first.
8
9	## Principles
10
11	1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
12	2. Confirm before proceeding. Wait for the user to agree before moving to the next step.
13	3. Don't read files until you need them. Only read reference files when you've reached the step that requires them.
14	4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.
15	5. No narration. Share outcomes and ask questions. Keep responses short.
16	6. No repetition. If you said something before a tool call, don't repeat it after.
17
18	## Scope
19
20	This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.
21
22	Tell the user when the skill is activated:
23
24	> "I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."
25
26	If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.
27
28	## Evaluation Types
29
30	There are two evaluation types:
31
32	- LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)
33	- Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.
34
35	## Workflow
36
37	### Step 1: Determine evaluation type
38
39	Do you already know which evaluation type to use?
40
41	Check conversation history, plan.md, workflow_state.json, or anything else you've already read.
42
43	If yes: confirm with the user.
44
45	> "It sounds like you want to run [evaluation type]. Is that right?"
46
47	⏸ Wait for confirmation. If confirmed → go to Step 2.
48
49	If no: ask.
50
51	> "What kind of evaluation would you like to run? I support:
52	>
53	> 1. LLM-as-Judge — an LLM grades your model's responses
54	> 2. Custom Scorer — programmatic scoring (math, code, or your own logic)
55	>
56	> Pick one, or say 'help me decide' if you're not sure."
57
58	⏸ Wait for user.
59
60	- If user picks one → go to Step 2.
61	- If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read `references/evaluation-type-guide.md` and follow its instructions. It will guide the user to a choice and then return here.
62	You MUST NEVER make a recommendation to the user on eval type without reading `references/evaluation-type-guide.md`.
63
64	### Step 2: Validate and hand off to evaluation workflow
65
66	Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.
67
68	#### LLM-as-Judge validation
69
70	1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):
71	- If you have the training job name or ARN, use the AWS MCP tool `list-tags` on the training job ARN and look for the `sagemaker-studio:jumpstart-model-id` tag. Contains "nova" → Nova. Anything else → OSS.
72	- If you have a Model Package ARN, use the AWS MCP tool `describe-model-package` and check the model description or source tags.
73	- If neither is available, ask the user.
74	2. Does the user have an evaluation dataset? LLM-as-Judge requires one.
75
76	#### Custom Scorer validation
77
78	1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)
79
80	---
81
82	If validation fails, tell the user which requirement(s) aren't met and offer alternatives:
83
84	> "[Evaluation type] won't work because [reason]."
85
86	If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:
87
88	> "Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."
89
90	If the failure reason is something else, offer to help them pick a different evaluation type.
91
92	⏸ Wait for user.
93
94	If they say they do want help choosing a different eval type → read `references/evaluation-type-guide.md`.
95
96	If validation passes, read the corresponding reference file:
97
98	\| User

People who liked this also viewed

6 picks

Type

model-evaluation

843stars132forks

View on GitHub

How to install model-evaluation?

awslabs/agent-plugins/model-evaluation

$npx -y skills add awslabs/agent-plugins --skill model-evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Model Evaluation
2
3	Generate code that evaluates a SageMaker model.
4
5	## Prerequisites
6
7	- The SDK environment has been verified (SDK version, region, execution role). If not done, activate the `sdk-getting-started` skill first.
8
9	## Principles
10
11	1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn.
12	2. Confirm before proceeding. Wait for the user to agree before moving to the next step.
13	3. Don't read files until you need them. Only read reference files when you've reached the step that requires them.
14	4. Don't ask what you already know. If the answer is in conversation history, workflow_state.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask.
15	5. No narration. Share outcomes and ask questions. Keep responses short.
16	6. No repetition. If you said something before a tool call, don't repeat it after.
17
18	## Scope
19
20	This skill supports the evaluation feature for SageMaker Serverless Model Customization. It can evaluate any base or fine-tuned model supported by SageMaker serverless model customization — both OSS models (Llama, Mistral, Qwen, etc.) and Nova models.
21
22	Tell the user when the skill is activated:
23
24	> "I can help evaluate any base or fine-tuned model supported by SageMaker serverless model customization."
25
26	If the user requests help evaluating a model that isn't supported by SageMaker serverless model customization, explain that it is not supported by this skill.
27
28	## Evaluation Types
29
30	There are two evaluation types:
31
32	- LLM-as-Judge — an LLM grades your model's responses. (OSS models only — not supported for Nova.)
33	- Custom Scorer — programmatic evaluation via Lambda function (includes built-in math and code scorers). Works with both OSS and Nova models.
34
35	## Workflow
36
37	### Step 1: Determine evaluation type
38
39	Do you already know which evaluation type to use?
40
41	Check conversation history, plan.md, workflow_state.json, or anything else you've already read.
42
43	If yes: confirm with the user.
44
45	> "It sounds like you want to run [evaluation type]. Is that right?"
46
47	⏸ Wait for confirmation. If confirmed → go to Step 2.
48
49	If no: ask.
50
51	> "What kind of evaluation would you like to run? I support:
52	>
53	> 1. LLM-as-Judge — an LLM grades your model's responses
54	> 2. Custom Scorer — programmatic scoring (math, code, or your own logic)
55	>
56	> Pick one, or say 'help me decide' if you're not sure."
57
58	⏸ Wait for user.
59
60	- If user picks one → go to Step 2.
61	- If user indicates uncertainty, by saying something like "help me decide," "whatever you think," "I'm not sure" → read `references/evaluation-type-guide.md` and follow its instructions. It will guide the user to a choice and then return here.
62	You MUST NEVER make a recommendation to the user on eval type without reading `references/evaluation-type-guide.md`.
63
64	### Step 2: Validate and hand off to evaluation workflow
65
66	Before reading the reference file, validate that the chosen evaluation type is compatible with the user's situation. You may already know these answers from conversation context — don't ask if you don't need to.
67
68	#### LLM-as-Judge validation
69
70	1. What model type are we evaluating? LLM-as-Judge is not supported for Nova models. To determine model type (if you don't already know it):
71	- If you have the training job name or ARN, use the AWS MCP tool `list-tags` on the training job ARN and look for the `sagemaker-studio:jumpstart-model-id` tag. Contains "nova" → Nova. Anything else → OSS.
72	- If you have a Model Package ARN, use the AWS MCP tool `describe-model-package` and check the model description or source tags.
73	- If neither is available, ask the user.
74	2. Does the user have an evaluation dataset? LLM-as-Judge requires one.
75
76	#### Custom Scorer validation
77
78	1. Does the user have an evaluation dataset? Custom Scorer requires one. (Works with both OSS and Nova models, though for Nova only custom lambdas are supported.)
79
80	---
81
82	If validation fails, tell the user which requirement(s) aren't met and offer alternatives:
83
84	> "[Evaluation type] won't work because [reason]."
85
86	If the failure reason was lack of an eval dataset, there's nothing we can do. Inform the user:
87
88	> "Unfortunately all of the supported eval types require an eval dataset. I can't help you with model evaluation."
89
90	If the failure reason is something else, offer to help them pick a different evaluation type.
91
92	⏸ Wait for user.
93
94	If they say they do want help choosing a different eval type → read `references/evaluation-type-guide.md`.
95
96	If validation passes, read the corresponding reference file:
97
98	\| User

People who liked this also viewed

6 picks

Type