Evaluating LLM Applications: Test Sets, Graders and Regression Checks

Most LLM applications ship with a handful of examples someone tried by hand. That is fine for a prototype and dangerous afterwards, because LLM systems change constantly: prompts get edited, retrieval settings get tuned, the provider releases a new model version, a dependency changes how documents are parsed. Without a repeatable way to measure quality, each of these is a guess, and regressions are discovered by users.

An evaluation (eval) harness is the LLM equivalent of a test suite. It will not be as crisp as unit tests, but it turns "it seems better" into a number you can compare across changes. It is also the most direct answer to the question from where AI actually pays off: what happens when it's wrong, and how would we know?

Start with the test set

The test set matters more than the tooling. A few hundred well-chosen cases beat thousands of synthetic ones that do not resemble real traffic.

• Source from reality. Pull real inputs from logs, support tickets or the workflow being automated (with appropriate redaction).
• Cover the distribution, then the edges. Include common cases in proportion, then deliberately add hard ones: ambiguous inputs, missing information, adversarial phrasing, other languages if you serve them.
• Include cases where the right answer is to refuse or escalate. A system that never says "I don't know" will score well on easy sets and badly in production.
• Write down the expected outcome as precisely as the task allows: an exact label, required facts, a reference answer, or a rubric.
• Grow it continuously. Every production failure that gets reported becomes a new test case.

{"id": "triage-0142", "input": "Charged twice for my March invoice, need one refunded", "expected": {"category": "billing", "priority": "normal", "route": "billing_queue"}, "tags": ["billing", "duplicate_charge"]}
{"id": "rag-0087", "input": "How many days of parental leave do contractors get?", "expected": {"must_include": ["not eligible"], "must_cite": ["hr-policy-2026-04"]}, "tags": ["rag", "negative"]}
{"id": "rag-0091", "input": "What is the travel per diem for Singapore?", "expected": {"behaviour": "refuse_not_in_docs"}, "tags": ["rag", "out_of_scope"]}

Tags let you report quality per slice. An overall score that holds steady can hide a collapse in one category.

Choosing graders

Use the cheapest grader that measures what you care about. Most mature harnesses combine all three types.

For model-graded evals, give the judge a specific rubric and ask for a structured verdict per criterion rather than a single 1–10 score. Then check it: have people label a sample of a hundred or so outputs and measure how often the judge agrees with them. If agreement is poor, fix the rubric before trusting the numbers. Re-check whenever you change the judge model.

A minimal harness

You do not need a platform to start. A script that runs the test set through the application and applies graders is enough, as long as it runs the real pipeline (same prompts, retrieval and post-processing), not a simplified copy.

import json, statistics
from app import pipeline              # the production code path
from graders import exact_fields, cites, judge_faithfulness

GRADERS = {
    "category": exact_fields,          # code-based
    "citations": cites,                # code-based
    "faithful": judge_faithfulness,    # model-graded, rubric in graders/rubrics/
}

def run(path: str, config: dict) -> dict:
    results = []
    for line in open(path):
        case = json.loads(line)
        output = pipeline.run(case["input"], config=config)
        scores = {name: g(case, output) for name, g in GRADERS.items() if g.applies(case)}
        results.append({"id": case["id"], "tags": case["tags"], "scores": scores, "output": output})
    return summarise(results)        # pass rate per grader, per tag, plus failures list

if __name__ == "__main__":
    baseline = json.load(open("evals/baseline.json"))
    current = run("evals/cases.jsonl", config=load_config())
    report_diff(baseline, current)    # fail the build on regressions beyond tolerance

Store every run's outputs, not just the scores. When a number moves, the first thing anyone wants is to read the cases that changed.

Make it a regression gate

• Run the suite on every change to prompts, retrieval settings, tools, model version or parsing code, the same way you run tests on code changes.
• Compare against a stored baseline and fail on regressions beyond a tolerance you agree in advance, per slice as well as overall.
• LLM outputs vary between runs. For important decisions, run cases more than once or use a larger set, and do not chase differences smaller than the run-to-run noise.
• Pin model versions in production where your provider allows it, and treat a model upgrade as a change that must pass the suite like any other.

Offline evals are not enough

A test set represents yesterday's traffic. Production monitoring closes the loop:

• Sample production outputs for periodic human or model-graded review.
• Track proxy signals: escalation rates, user corrections, refusals, thumbs-down, downstream rework.
• Watch input drift: new topics, products or document types the test set does not cover.
• Feed confirmed failures back into the test set.

Logged inputs and outputs are often sensitive; apply the controls in AI data privacy and governance to eval data as well.

Specific notes by application type

• RAG: score retrieval (did the expected document appear in the top results?) separately from the answer. See RAG for enterprise documents.
• Agents: score the outcome and the trajectory, including tool calls made, permissions respected and steps used. See AI agents in the enterprise.
• Cost changes: when routing traffic to a smaller model or trimming prompts, the eval suite is what tells you whether the saving is safe. See LLM cost control.

Key takeaways

• Build the test set from real inputs, include refusal and edge cases, and tag by slice.
• Prefer code-based graders; validate any LLM judge against human labels.
• Run evals on every change and gate releases on regressions, including model upgrades.
• Pair offline evals with production sampling and feed failures back in.

Evaluation design is part of every AI enablement engagement HashTech runs, because it is what makes the later move to production measurable.