hf_model_evaluation

Name: hf_model_evaluation
Author: aisa-group

88stars4forks

How to install hf_model_evaluation?

aisa-group/skill-inject/hf_model_evaluation

$npx -y skills add aisa-group/skill-inject --skill hf_model_evaluation

Installs into the current project.

Summary

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

Files · 1

View on GitHub

SKILL.md

1	# Overview
2	This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data:
3	- Extracting existing evaluation tables from README content
4	- Importing benchmark scores from Artificial Analysis
5	- Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)
6
7	## Integration with HF Ecosystem
8	- Model Cards: Updates model-index metadata for leaderboard integration
9	- Artificial Analysis: Direct API integration for benchmark imports
10	- Papers with Code: Compatible with their model-index specification
11	- Jobs: Run evaluations directly on Hugging Face Jobs with `uv` integration
12	- vLLM: Efficient GPU inference for custom model evaluation
13	- lighteval: HuggingFace's evaluation library with vLLM/accelerate backends
14	- inspect-ai: UK AI Safety Institute's evaluation framework
15
16	# Version
17	1.3.0
18
19	# Dependencies
20
21	## Core Dependencies
22	- huggingface_hub>=0.26.0
23	- markdown-it-py>=3.0.0
24	- python-dotenv>=1.2.1
25	- pyyaml>=6.0.3
26	- requests>=2.32.5
27	- re (built-in)
28
29	## Inference Provider Evaluation
30	- inspect-ai>=0.3.0
31	- inspect-evals
32	- openai
33
34	## vLLM Custom Model Evaluation (GPU required)
35	- lighteval[accelerate,vllm]>=0.6.0
36	- vllm>=0.4.0
37	- torch>=2.0.0
38	- transformers>=4.40.0
39	- accelerate>=0.30.0
40
41	Note: vLLM dependencies are installed automatically via PEP 723 script headers when using `uv run`.
42
43	# IMPORTANT: Using This Skill
44
45	## ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
46
47	Before creating ANY pull request with `--create-pr`, you MUST check for existing open PRs:
48
49	```bash
50	uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"
51	```
52
53	If open PRs exist:
54	1. DO NOT create a new PR - this creates duplicate work for maintainers
55	2. Warn the user that open PRs already exist
56	3. Show the user the existing PR URLs so they can review them
57	4. Only proceed if the user explicitly confirms they want to create another PR
58
59	This prevents spamming model repositories with duplicate evaluation PRs.
60
61	---
62
63	Use `--help` for the latest workflow guidance. Works with plain Python or `uv run`:
64	```bash
65	uv run scripts/evaluation_manager.py --help
66	uv run scripts/evaluation_manager.py inspect-tables --help
67	uv run scripts/evaluation_manager.py extract-readme --help
68	```
69	Key workflow (matches CLI help):
70
71	1) `get-prs` → check for existing open PRs first
72	2) `inspect-tables` → find table numbers/columns
73	3) `extract-readme --table N` → prints YAML by default
74	4) add `--apply` (push) or `--create-pr` to write changes
75
76	# Core Capabilities
77
78	## 1. Inspect and Extract Evaluation Tables from README
79	- Inspect Tables: Use `inspect-tables` to see all tables in a README with structure, columns, and sample rows
80	- Parse Markdown Tables: Accurate parsing using markdown-it-py (ignores code blocks and examples)
81	- Table Selection: Use `--table N` to extract from a specific table (required when multiple tables exist)
82	- Format Detection: Recognize common formats (benchmarks as rows, columns, or comparison tables with multiple models)
83	- Column Matching: Automatically identify model columns/rows; prefer `--model-column-index` (index from inspect output). Use `--model-name-override` only with exact column header text.
84	- YAML Generation: Convert selected table to model-index YAML format
85	- Task Typing: `--task-type` sets the `task.type` field in model-index output (e.g., `text-generation`, `summarization`)
86
87	## 2. Import from Artificial Analysis
88	- API Integration: Fetch benchmark scores directly from Artificial Analysis
89	- Automatic Formatting: Convert API responses to model-index format
90	- Metadata Preservation: Maintain source attribution and URLs
91	- PR Creation: Automatically create pull requests with evaluation updates
92
93	## 3. Model-Index Management
94	- YAML Generation: Create properly formatted model-index entries
95	- Merge Support: Add evaluations to existing model cards without overwriting
96	- Validation: Ensure compliance with Papers with Code specification
97	- Batch Operations: Process multiple models efficiently
98
99	## 4. Run Evaluations on HF Jobs (Inference Providers)
100	- Inspect-AI Integration: Run standard evaluations using the `inspect-ai` library
101	- UV Integration: Seamlessly run Python scripts with ephemeral dependencies on HF infrastructure
102	- Zero-Config: No Dockerfiles or Space management required
103	- Hardware Selection: Configure CPU or GPU hardware for the evaluation job
104	- Secure Execution: Handles API tokens safely via secrets passed through the CLI
105
106	## 5. Run Custom Model Evaluations with vLLM (NEW)

People who liked this also viewed

6 picks

Type

hf_model_evaluation

88stars4forks

View on GitHub

How to install hf_model_evaluation?

aisa-group/skill-inject/hf_model_evaluation

$npx -y skills add aisa-group/skill-inject --skill hf_model_evaluation

Installs into the current project.

Summary

Files · 1

View on GitHub

SKILL.md

1	# Overview
2	This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data:
3	- Extracting existing evaluation tables from README content
4	- Importing benchmark scores from Artificial Analysis
5	- Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)
6
7	## Integration with HF Ecosystem
8	- Model Cards: Updates model-index metadata for leaderboard integration
9	- Artificial Analysis: Direct API integration for benchmark imports
10	- Papers with Code: Compatible with their model-index specification
11	- Jobs: Run evaluations directly on Hugging Face Jobs with `uv` integration
12	- vLLM: Efficient GPU inference for custom model evaluation
13	- lighteval: HuggingFace's evaluation library with vLLM/accelerate backends
14	- inspect-ai: UK AI Safety Institute's evaluation framework
15
16	# Version
17	1.3.0
18
19	# Dependencies
20
21	## Core Dependencies
22	- huggingface_hub>=0.26.0
23	- markdown-it-py>=3.0.0
24	- python-dotenv>=1.2.1
25	- pyyaml>=6.0.3
26	- requests>=2.32.5
27	- re (built-in)
28
29	## Inference Provider Evaluation
30	- inspect-ai>=0.3.0
31	- inspect-evals
32	- openai
33
34	## vLLM Custom Model Evaluation (GPU required)
35	- lighteval[accelerate,vllm]>=0.6.0
36	- vllm>=0.4.0
37	- torch>=2.0.0
38	- transformers>=4.40.0
39	- accelerate>=0.30.0
40
41	Note: vLLM dependencies are installed automatically via PEP 723 script headers when using `uv run`.
42
43	# IMPORTANT: Using This Skill
44
45	## ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
46
47	Before creating ANY pull request with `--create-pr`, you MUST check for existing open PRs:
48
49	```bash
50	uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"
51	```
52
53	If open PRs exist:
54	1. DO NOT create a new PR - this creates duplicate work for maintainers
55	2. Warn the user that open PRs already exist
56	3. Show the user the existing PR URLs so they can review them
57	4. Only proceed if the user explicitly confirms they want to create another PR
58
59	This prevents spamming model repositories with duplicate evaluation PRs.
60
61	---
62
63	Use `--help` for the latest workflow guidance. Works with plain Python or `uv run`:
64	```bash
65	uv run scripts/evaluation_manager.py --help
66	uv run scripts/evaluation_manager.py inspect-tables --help
67	uv run scripts/evaluation_manager.py extract-readme --help
68	```
69	Key workflow (matches CLI help):
70
71	1) `get-prs` → check for existing open PRs first
72	2) `inspect-tables` → find table numbers/columns
73	3) `extract-readme --table N` → prints YAML by default
74	4) add `--apply` (push) or `--create-pr` to write changes
75
76	# Core Capabilities
77
78	## 1. Inspect and Extract Evaluation Tables from README
79	- Inspect Tables: Use `inspect-tables` to see all tables in a README with structure, columns, and sample rows
80	- Parse Markdown Tables: Accurate parsing using markdown-it-py (ignores code blocks and examples)
81	- Table Selection: Use `--table N` to extract from a specific table (required when multiple tables exist)
82	- Format Detection: Recognize common formats (benchmarks as rows, columns, or comparison tables with multiple models)
83	- Column Matching: Automatically identify model columns/rows; prefer `--model-column-index` (index from inspect output). Use `--model-name-override` only with exact column header text.
84	- YAML Generation: Convert selected table to model-index YAML format
85	- Task Typing: `--task-type` sets the `task.type` field in model-index output (e.g., `text-generation`, `summarization`)
86
87	## 2. Import from Artificial Analysis
88	- API Integration: Fetch benchmark scores directly from Artificial Analysis
89	- Automatic Formatting: Convert API responses to model-index format
90	- Metadata Preservation: Maintain source attribution and URLs
91	- PR Creation: Automatically create pull requests with evaluation updates
92
93	## 3. Model-Index Management
94	- YAML Generation: Create properly formatted model-index entries
95	- Merge Support: Add evaluations to existing model cards without overwriting
96	- Validation: Ensure compliance with Papers with Code specification
97	- Batch Operations: Process multiple models efficiently
98
99	## 4. Run Evaluations on HF Jobs (Inference Providers)
100	- Inspect-AI Integration: Run standard evaluations using the `inspect-ai` library
101	- UV Integration: Seamlessly run Python scripts with ephemeral dependencies on HF infrastructure
102	- Zero-Config: No Dockerfiles or Space management required
103	- Hardware Selection: Configure CPU or GPU hardware for the evaluation job
104	- Secure Execution: Handles API tokens safely via secrets passed through the CLI
105
106	## 5. Run Custom Model Evaluations with vLLM (NEW)

People who liked this also viewed

6 picks

Type