promptfoo-action
starsforks
How to install promptfoo-action?
promptfoo/promptfoo-action
$git clone https://github.com/promptfoo/promptfoo-actionThis GitHub Action uses Promptfoo to evaluate prompts when monitored files change.
README.md
| 1 | # GitHub Action for LLM Prompt Evaluation |
| 2 | |
| 3 | This GitHub Action uses [Promptfoo](https://www.promptfoo.dev) to evaluate |
| 4 | prompts when monitored files change. |
| 5 | |
| 6 | On pull requests, the action evaluates the current checkout and posts a summary |
| 7 | comment with pass/fail counts and, when sharing is enabled, a link to the |
| 8 | Promptfoo web viewer: |
| 9 | |
| 10 | <img width="650" alt="pull request llm eval" src="https://github.com/typpo/promptfoo-action/assets/310310/ec75fb39-c6b1-4395-9e41-6d66a7bf8657"/> |
| 11 | |
| 12 | The web viewer lets you inspect the evaluation results: |
| 13 | |
| 14 | <img width="650" alt="promptfoo web viewer" src="https://github.com/typpo/promptfoo-action/assets/310310/d0ef0497-0c1a-4886-b115-1ee92680891b"/> |
| 15 | |
| 16 | ## Supported Events |
| 17 | |
| 18 | This action supports multiple GitHub event types: |
| 19 | |
| 20 | - **Pull Request** (`pull_request`, `pull_request_target`) - Uses the pull |
| 21 | request file list to select matching prompt files and posts a PR comment. |
| 22 | - **Push** (`push`) - Uses the before/after commit SHAs to select matching |
| 23 | prompt files and writes a workflow summary. |
| 24 | - **Manual Trigger** (`workflow_dispatch`) - Uses a supplied file list or a git |
| 25 | comparison against a base ref and writes a workflow summary. |
| 26 | |
| 27 | If change detection is unavailable, the action evaluates all files matching the |
| 28 | configured `prompts` globs. The action evaluates only the current checkout; it |
| 29 | does not run separate base and head evaluations. |
| 30 | |
| 31 | For `pull_request_target`, use extra care with checkout configuration and |
| 32 | credentials. Do not execute untrusted pull request code with a privileged token. |
| 33 | |
| 34 | ## Configuration |
| 35 | |
| 36 | The action can be configured using the following inputs: |
| 37 | |
| 38 | | Parameter | Description | Required | |
| 39 | | --- | --- | --- | |
| 40 | | `config` | Promptfoo configuration path, relative to `working-directory` unless absolute. | Yes | |
| 41 | | `github-token` | GitHub token used to list PR files and post PR comments. | Yes | |
| 42 | | `prompts` | Newline-separated prompt glob patterns, resolved from `working-directory`. Matching changed files are passed to Promptfoo with `--prompts`. If omitted, Promptfoo uses the prompts in `config`. | No | |
| 43 | | `working-directory` | Base directory for the Promptfoo process and relative config, prompt, environment, and cache paths. Defaults to `.`. | No | |
| 44 | | `cache-path` | Promptfoo disk-cache directory. Relative paths are resolved from `working-directory`. | No | |
| 45 | | `promptfoo-version` | Version or dist-tag used by `npx promptfoo@<version>`. Defaults to `latest`. | No | |
| 46 | | `no-share` | Pass `--no-share`, overriding config-level sharing. Defaults to `false`. | No | |
| 47 | | `use-config-prompts` | Do not override config prompts with changed files matched by `prompts`. Defaults to `false`. | No | |
| 48 | | `env-files` | Comma-separated `.env` paths loaded in order from `working-directory`. Later files override earlier files. | No | |
| 49 | | `fail-on-threshold` | Required suite pass percentage from 0 to 100. | No | |
| 50 | | `max-concurrency` | Value passed to Promptfoo's `--max-concurrency`. Defaults to `4`. | No | |
| 51 | | `no-table` | Pass `--no-table`. Defaults to `false`. | No | |
| 52 | | `no-progress-bar` | Pass `--no-progress-bar`. Defaults to `false`. | No | |
| 53 | | `no-cache` | Pass `--no-cache` so Promptfoo does not read or write cached evaluation results. Defaults to `false`. | No | |
| 54 | | `disable-comment` | Disable posting comments to the PR. Defaults to `false`. Non-PR workflow summaries are unaffected. | No | |
| 55 | | `workflow-files` | Newline-separated changed-file list for `workflow_dispatch`. Takes precedence over workflow-level `files`. | No | |
| 56 | | `workflow-base` | Base branch, tag, full commit SHA, or supported `HEAD` revision for `workflow_dispatch`. Takes precedence over workflow-level `base`; defaults to `HEAD~1`. | No | |
| 57 | | `repeat` | Number of times Promptfoo runs each test. Must be at least `2`; omit it to run once. | No | |
| 58 | | `repeat-min-pass` | Minimum passes required for each repeated test. Requires `repeat` and cannot exceed it. | No | |
| 59 | | `force-run` | Evaluate even when change detection finds no relevant files. Defaults to `false`. | No | |
| 60 | | `debug` |
People who liked this also viewed
6 picksType
- promptpexTest Generation for PromptsPromptsMay 2026
- langfuse🪢 Open source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. Integrates with OpenTelemetry, LangChain, OpenAI SDK, LiteLLM, and more. 🍊YC W23 PromptsJul 2026
- promptflowBuild high-quality LLM apps - from prototyping, testing to production deployment and monitoring.PromptsJul 2026
- pezzo🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.PromptsMar 2026
- ChainForgeAn open-source visual programming environment for battle-testing prompts to LLMs.PromptsJun 2026
- PromptFuzzPromtFuzz is an automated tool that generates high-quality fuzz drivers for libraries via a fuzz loop constructed on mutating LLMs' prompts.PromptsMay 2026
