Add evals to your pull requests in 10 minutes

This guide adds an eval check to your pull requests. When a change makes your model worse, the pull request shows a failing check, Evalban files a bug with the examples that broke, and the issue stays out of Done.

You need a GitHub repository, an eval script you can run from the command line, and an Evalban workspace.

1. Create the experiment

In Evalban, open Models & evals and click New experiment. Give it a name, the metric your eval reports (for example accuracy), and a baseline: the score it must not fall below. Its id, like support-bot, is what your code sends.

If the experiment should gate an issue, pick it now. That issue will wait in Evaluating after its pull request merges, until a run passes.

2. Create an API key and save two secrets

In Evalban, open API keys and create a key. Copy it; it's shown once. The same page shows your EVALBAN_URL.

In GitHub, open your repository's Settings → Secrets and variables → Actions and add two repository secrets: EVALBAN_API_KEY and EVALBAN_URL.

3. Have your eval write a results file

Your eval keeps doing what it does. At the end, write the score and any failing examples to a JSON file:

import json

results = run_my_eval()          # your existing eval code

with open("eval-results.json", "w") as f:
    json.dump({
        "score": results.accuracy,
        "failures": [
            {"input": c.question, "expected": c.expected, "output": c.answer}
            for c in results.failed[:100]
        ],
    }, f)

Already using promptfoo? Skip this step and point the action at promptfoo's output with results-format: promptfoo. JSON Lines and CSV work too.

4. Add the workflow

Create .github/workflows/evals.yml:

name: Evals
on:
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python run_evals.py          # writes eval-results.json

      - name: Send to Evalban
        uses: evalban-dev/eval-action@v1
        with:
          api-key: ${{ secrets.EVALBAN_API_KEY }}
          url: ${{ secrets.EVALBAN_URL }}
          experiment: support-bot
          results-file: eval-results.json
          fail-on-regression: true

The version defaults to the commit's short SHA, so every run in Evalban points back to the exact change that produced it. Add cost-per-call-usd and latency-p95-ms if you track them, and Evalban flags runs that go over budget.

5. Open a pull request

Put the issue key in the branch name or title, like sup-12-refund-answers. When the workflow runs:

• It passes: the check is green, and the run appears on the experiment's chart and timeline.
• It regressed: the check fails, Evalban files a bug with the failing examples, posts to Slack if it's connected, and the action's outputs include the bug's key.

Connect GitHub in Evalban (GitHub & Slack) and the pull request also moves the issue: a draft to In progress, an open pull request to In review, and a merge to Done, or to Evaluating until its evals pass.

Tips

• Keep the pull-request eval fast. Run a representative sample on every pull request, and the full set nightly with a scheduled workflow.
• Set a tolerance. Small score changes between runs are often noise. A tolerance in the experiment's settings stops those from filing bugs.
• Save failures as cases. From a bug in Evalban, save the failing example to the eval set. Your script can fetch the set with get_eval_cases(), so a fixed failure can't quietly come back.

Everything the action and API accept is in the developer docs.