Send your evals to Evalban.

Quickstart

1. Create an experiment. In Evalban, open Models & evals and click New experiment. Give it a name, the metric (for example accuracy), a baseline (for example 0.85) and, if you like, the issue it gates. Note its id, like listing-classifier.
2. Create an API key. Open API keys, click Create key and copy it. It starts with sk_evalban_ and is shown once. That page also shows your EVALBAN_URL.
3. Save both as secrets wherever your evals run: EVALBAN_API_KEY and EVALBAN_URL.
4. Send a run at the end of your eval script:

pip install evalban

import os
from evalban import Evalban

client = Evalban(api_key=os.environ["EVALBAN_API_KEY"], url=os.environ["EVALBAN_URL"])
client.log_run("listing-classifier", 0.871, version="ft-1027")

That's it. Open the experiment in Evalban to see the run, the trend and the verdict.

How Evalban decides

• Baseline. Each experiment has a score it must not drop below (or rise above, for metrics where lower is better, like latency). A run below the baseline is a regression.
• Tolerance. Drops smaller than the experiment's tolerance count as within noise, not a regression.
• Confirmation. You can ask for 2 or more bad runs in a row before a bug is filed. Until then the run is unconfirmed.
• Slices and extra metrics. Send a score per slice (like refunds) and they're checked against the baseline too, or against their own line if you set one.
• Budgets. Set a maximum cost per call or p95 latency; going over is flagged on the run.
• Gating. If the experiment gates an issue, that issue waits in Evaluating after its pull request merges, and moves to Done by itself when a run passes.
• Bugs. On a regression, Evalban files a bug with the failing examples attached and tells Slack. If an example saved from a fixed bug fails again, that bug is reopened.

Every run returns a verdict: pass, within_noise, suspect (unconfirmed drop) or regressed.

Python SDK

The evalban package has no dependencies beyond the Python standard library.

import os
from evalban import Evalban

client = Evalban(api_key=os.environ["EVALBAN_API_KEY"], url=os.environ["EVALBAN_URL"])

res = client.log_run("listing-classifier", 0.871,
    version="ft-1027",
    failures=[{"input": "Grey sofa", "expected": "Furniture", "output": "Outdoor"}],
    slices={"furniture": 0.74},
    cost_per_call_usd=0.0004, latency_p95_ms=168)

if res["gate"] == "regressed":
    print("Evalban filed", res["bug_key"])

Fetch the eval set (including cases saved from fixed bugs) and send production feedback:

cases = client.get_eval_cases("listing-classifier")

client.send_feedback("reply-drafts", [{"kind": "negative", "input": "...", "output": "..."}])

Command line

The package also installs an evalban command. It reads EVALBAN_API_KEY and EVALBAN_URL from the environment and exits with status 1 when the run regressed, so it can fail a CI job.

evalban log --experiment listing-classifier --file results.json --format promptfoo --version ft-1027

Formats: evalban, promptfoo, jsonl, csv.

GitHub Action

Send a run from any workflow with evalban-dev/eval-action.

- name: Send evals to Evalban
  uses: evalban-dev/eval-action@v1
  with:
    api-key: ${{ secrets.EVALBAN_API_KEY }}
    url: ${{ secrets.EVALBAN_URL }}
    experiment: listing-classifier
    results-file: eval-results.json      # or: score: 0.871
    results-format: promptfoo            # evalban, promptfoo, jsonl or csv
    fail-on-regression: true

Outputs: gate, verdict, bug-key, run-number, returned (reopened bugs) and over-budget.

API: send a run

POST https://vegwydmbxmhzatcrorin.supabase.co/functions/v1/eval-ingest

Send your key as Authorization: Bearer sk_evalban_… and a JSON body.

curl -X POST https://vegwydmbxmhzatcrorin.supabase.co/functions/v1/eval-ingest \
  -H "Authorization: Bearer $EVALBAN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"experiment": "listing-classifier", "score": 0.871, "version": "ft-1027"}'

Returns 201 with the result:

{
  "run_number": 5, "score": 0.871, "baseline": 0.88,
  "verdict": "regressed", "gate": "regressed",
  "reasons": [{"kind": "primary", "name": "macro F1", "value": 0.871, "threshold": 0.88}],
  "bug_key": "ATL-12", "bug_filed": true, "failing_examples": 1,
  "returned": [], "over_budget": []
}

API: get the eval set

GET https://vegwydmbxmhzatcrorin.supabase.co/functions/v1/eval-cases?experiment=listing-classifier

Returns {"cases": [{"key", "input", "expected"}]}, including cases saved from bugs. Run them in your eval script and send the results back as examples, so a fixed bug can't quietly return.

API: send feedback

POST https://vegwydmbxmhzatcrorin.supabase.co/functions/v1/feedback-ingest

Send thumbs-up/down and flagged answers from production, or forward them from your logs. When negative feedback spikes, Evalban files a bug and tells Slack.

{
  "experiment": "reply-drafts",
  "events": [{"kind": "negative", "input": "...", "output": "...", "version": "prompt v6", "trace_id": "abc123"}]
}

kind is positive, negative or flagged. Up to 500 events per request. Returns how many were received and the negative rate over the last hour.

Feedback widget

Collect 👍/👎 from your users without writing a backend. In Evalban, open the experiment, scroll to Feedback widget, add the websites it will run on, and copy the code. Its key starts with pk_evalban_, can only send feedback for that one experiment, and only works from the websites you listed, so it's safe in a web page.

<script src="https://app.evalban.com/widget.js" data-key="pk_evalban_..." async></script>

<!-- Under each AI answer: -->
<div data-evalban-feedback
     data-input="the user's question"
     data-output="the model's answer"
     data-version="prompt v6"></div>

Answers added to the page later get the buttons too. To use your own buttons instead:

Evalban.feedback({ kind: 'negative', input: question, output: answer, version: 'prompt v6', trace_id: id })

Errors and limits

Errors come back as {"error": "…"} with a message you can show as is. Questions or something missing? Email [email protected].