Test and evaluate AI agents and LLM outputs using code-first evaluation framework with strong typing. Use when the user wants to: (1) Create evaluation datasets with test cases for AI agents, (2) Define evaluators (deterministic, LLM-as-Judge, custom, or span-based), (3) Run evaluations and generate reports, (4) Compare model performance across experiments, (5) Integrate evaluations with Pydantic AI agents, (6) Set up observability with Logfire, (7) Generate test datasets using LLMs, (8) Implement regression testing for AI systems.
How to install pydantic-evals?
fuenfgeld/pydantic-ai-skills/pydantic-evals
$npx -y skills add fuenfgeld/pydantic-ai-skills --skill pydantic-evalsInstalls into the current project.
›Prefer a prompt? Paste this to your agent
Run `npx skills use "https://github.com/fuenfgeld/pydantic-ai-skills" --skill "fuenfgeld/pydantic-ai-skills/pydantic-evals"` and follow the generated skill instructions now. Read its complete output, redirecting it to a temporary file first if necessary. Resolve relative paths from the supporting-files directory it provides.
Use the skills in "https://github.com/fuenfgeld/pydantic-ai-skills" that are relevant to the current task. Run `npx skills add "https://github.com/fuenfgeld/pydantic-ai-skills"` and select the relevant skills, then follow their instructions.
Related
6 picksType
- azure-kustoQuery and analyze data in Azure Data Explorer (Kusto/ADX) using KQL for log analytics, telemetry, and time series analysis.Jul 2026
- analyticsWhen the user wants to set up, improve, or audit analytics tracking and measurement.Jul 2026
- firecrawl-dashboard-reportingPull metrics from analytics dashboards and internal web tools with Firecrawl browser.Jun 2026
- data-storytellingTransform data into compelling narratives using visualization, context, and persuasive structure.Jul 2026
- data-visualizationCreate effective data visualizations with Python (matplotlib, seaborn, plotly).Jul 2026
- extension-oqlMake a canister's data queryable by the Caffeine Data Intelligence agent.Jul 2026
