LLM Evaluation Guide: Is Your Chatbot Actually Working?

The difference between a chatbot that works in the demo and a chatbot that works in production is a test set. We put that claim first because every failed chatbot project we have seen in the field shares the same trait: nobody tied the sentence "it works well" to a number. The vendor demoed five questions, the team said "looks good", the system went live. Three weeks later a customer learned the wrong return policy.

This piece covers LLM evaluation (the industry shorthand is "evals") for decision-makers and technical teams together: how many examples you need, which metrics to measure, how far to trust one AI grading another, how to monitor in production, and what it all costs. For the wider map of infrastructure choices, see our non-technical leader's guide to AI infrastructure decisions; here we deal only with the measurement layer.

Why now? Because a 2026 academic study showed that prompt tweaks widely accepted as "best practice" can lower performance when applied without measurement. Every unmeasured "improvement" is a coin toss, and the team usually finds out which side it landed on from a customer.

Why is LLM evaluation different from classic software testing?

In classic software the same input always produces the same output, so a test asks "does it match the expected result". A language model can answer the same question in different sentences every time, and the answer reads fluently even when it is wrong. LLM evaluation therefore asks for judgment rather than matching: is the answer correct, grounded in a source, relevant to the question, safe, and what did it cost.

Second difference: testing the model and testing the application are separate things. A high benchmark score does not mean the model explains your return policy correctly. What concerns you is the system-level test: was the right document retrieved, the right tool called, the task completed. The core risk is a chatbot that is "fluent but wrong, ungrounded, unsafe or expensive"; a benchmark measures none of those.

Third difference: testing is never done once. The prompt, the model version and the document archive all change, and each change requires the test to run again. An eval is a gate on every change, not a document written once and shelved.

What is a golden set, and how many examples do you need?

A golden set is a versioned test collection compiled from real user questions and edge cases, with an expected answer or an acceptance criterion written down for each one. Start with around 50, aim for 200 or more diversified examples. Diversity matters more than the count: a 95 percent pass rate on 20 narrow examples tells you less than 88 percent on 200 varied ones.

That "95 on 20 versus 88 on 200" comparison comes from the DeepEval documentation; a vendor source, but the logic holds. Ten failures across 200 examples show which question type is weak; two failures across 20 show nothing. An open-source repository built on the eval patterns Anthropic published also starts at 50 examples, so "50 in the first week" is a practical target.

Each example holds the question, the context where relevant (in RAG systems, which document should have been retrieved), the expected answer or acceptance criterion, and labels (question type, difficulty, sensitivity). Before launch the examples come from questions your own staff would ask; after launch, from real conversation logs, especially where a user said "no, that's not right". Every production error goes back into the set; that is how it grows.

Which metrics matter for a RAG system?

For a chatbot that answers from company documents (RAG), there are three core metrics: faithfulness (is the answer grounded in the retrieved document, or invented), answer relevancy (does the answer actually address the question) and context precision (are the retrieved documents relevant, and are the relevant ones ranked on top). Together they are known as the "RAG triad", and each can fail independently.

Independent failure is the point. An answer can be faithful yet irrelevant: the right paragraph pasted onto the wrong question. It can be relevant yet unfaithful: made up from the model's own knowledge, the most dangerous case. If context precision is low, the problem is in retrieval; changing the prompt will not help, and the fix belongs on the embedding and search side. The metric tells you which layer is broken; a single "accuracy percentage" cannot.

One warning: Ragas and DeepEval compute metrics with the same name differently. Ragas checks whether each claim in the answer can be derived from the document; DeepEval looks for contradictions between output and document. A faithfulness score of 0.9 from the two tools is not the same thing. Pick one tool and track change over time; do not compare across tools. We covered RAG itself in our non-technical guide to RAG, and there is a separate guide on preventing hallucinations in production.

Can you trust an LLM to grade another LLM?

Partly. Using a language model as a judge (LLM-as-a-judge) is the only method that scales, but the judge has three well-documented biases: position bias (favoring the answer in a particular slot in a comparison), length bias (mistaking the longer answer for the better one) and self-preference (scoring its own generated text higher). These can be reduced, never eliminated.

The academic literature reports all three consistently; a 2026 paper questions judge reliability under the heading "coin-flip judge". The mitigations are known: randomize answer order and score in both directions, use a different model as judge than the one being evaluated, apply an explicit length penalty, and give the judge a concrete rubric instead of writing "be fair".

The real safeguard is calibration: a human labels a subset of the golden set (say 40 examples) and you measure agreement with the judge's scores. Low agreement means the rubric gets fixed; the judge is tested too. Skip this step and you are deciding on numbers from a judge of unknown reliability. A sensible division of labor follows: hard rules (banned phrases, format, length, JSON schema) are checked deterministically; the judge handles only what requires judgment (correctness, tone, completeness). The same labeled set later decides which questions a cheaper model can handle; model routing for cost is impossible without it.

How do you set up a regression gate and production monitoring?

Evals run in two places: a regression gate that runs automatically on every change before release, and monitoring that scores a sample of real traffic in production. The gate stops a change that falls below the threshold from going live; monitoring catches new failure types that are not in the golden set yet.

The gate works on the continuous-integration logic developers already know: when the prompt, model or retrieval layer changes, the golden set runs automatically, and a change that falls below the threshold is not merged. How do you pick the threshold? A 2024 academic paper covers it; in practice, teams take a point slightly above their baseline and raise it over time. What matters is that the threshold is written down and stands in front of the change.

Scoring every answer in production is unnecessary and expensive. A percentage of conversations is sampled, the judge scores them, and low-scoring conversations are clustered to find new failure patterns. Three cheap signals sit alongside this: thumbs up or down, the escalation rate to a human, and the rate of "I don't know" answers. A new version ships behind an A/B test or a shadow test (running in parallel, invisible to users) for a comparison against the old one.

Tools and cost: Ragas, DeepEval, Promptfoo, LangSmith, Braintrust

On the open-source side, Ragas and DeepEval provide the RAG metrics, and Promptfoo provides configuration-defined test flows that fit into continuous integration; all are free, and the only cost is the judge model's token fees. On the platform side, LangSmith charges per trace and Braintrust per score; both have free tiers for small teams.

The pricing detail rests on a single third-party source we did not verify against the official pages: LangSmith 5,000 traces free, then $2.50 per thousand; Braintrust 10,000 scores free, then $2.50 per thousand. Read the official page before you sign. Langfuse and Arize Phoenix cover the observability side, and the evaluation tool inside the Anthropic console is the lowest-barrier way to build a first golden set.

The real cost items are judge calls and human labeling; tool fees are small next to them. Each run of a 200-example golden set costs a few dollars at current prices; several runs a day add up to tens of dollars a month. Human calibration is half a day of one employee's time for 40 examples. The measurement layer is a few percent of a chatbot project's budget, and cheaper than one customer incident without it.

The Air Canada lesson: an untested chatbot in court

In February 2024 a consumer tribunal in Canada held Air Canada liable for incorrect information given by its chatbot. The bot had told a passenger to "buy a regular ticket and apply for the bereavement discount within 90 days"; the actual policy did not allow retroactive applications. The defense that "the chatbot is a separate legal entity" was rejected: it makes no difference, the tribunal said, whether the information came from a static page or a chatbot.

The amount was small, about 812 Canadian dollars. The lesson is large: the chatbot's policy statements (returns, prices, deadlines, conditions) are the company's statements. In eval terms, that means a dedicated golden-set category for policy questions: at least one example per return, price and deadline rule, an "outside the document" alarm on faithfulness, and a zero-tolerance threshold. Precedents differ by jurisdiction, but consumer bodies in most countries already treat a statement on a company web page as binding, and the direction is not hard to guess. Read this as a risk note, not legal advice.

Evaluating outside English: a retailer's returns assistant

An example from the field, company anonymous, numbers rounded. A 60-person online retailer built an assistant for return and shipping questions in its local language. The first golden set had 120 questions: 70 from real support logs, 30 policy questions (return window, shipping fee, damaged items) and 20 edge cases ("I opened it but never used it", "returns on a promotional item").

First measurement: faithfulness 0.81, answer relevancy 0.88, and 4 errors out of 30 on policy questions. The problem sat in retrieval: the promotional terms lived in a separate document that was never found. The documents were merged and search was fixed; faithfulness rose to 0.93 and policy errors dropped to zero, without touching a word of the prompt. For judge calibration, a support agent labeled 40 examples; agreement with the judge was 78 percent in the first round, because the rubric read the language's politeness formulas as "incomplete answer". Once corrected, agreement reached 90 percent.

Turkey spotlight: this retailer operated in Turkish, and the detail is a general warning for any language other than English. Off-the-shelf rubrics and metric prompts are written in English; without a golden set and a rubric in the local language, the judge ends up grading the language rather than your chatbot. Agglutinative languages such as Turkish, where one word can carry a whole clause, make this worse.

Frequently asked questions

How many examples do I need to test a chatbot?

Start with 50, aim for 200 or more, diversified. Do not trust a high percentage on a small set; 95 percent on 20 examples carries less information than 88 percent on 200.

What is the difference between faithfulness and answer relevancy?

Faithfulness is whether the answer is grounded in the retrieved document; relevancy is whether the answer addresses the question. One can be low while the other is high, and each points at a different layer.

Is it reliable to have an AI grade an AI?

Usable once calibrated. Position, length and self-preference biases are well known; use a different judge model, randomized order and an agreement check against 40 human-labeled examples.

What does evaluation cost?

The tools are largely open source; the real items are judge calls (a few dollars per run for 200 examples) and human labeling (half a day). A few percent of the project budget.

Is the company liable if the chatbot gives wrong information?

In the Air Canada ruling, yes: no different from a static page. Build a separate test category for policy questions with a zero-tolerance threshold.

So what should you actually do?

• Write a 50-example golden set in the first week; from real questions, with policy questions labeled separately.
• Measure the three RAG metrics separately; a single accuracy percentage cannot tell you which layer is broken.
• Calibrate the judge: 40 human-labeled examples, a different judge model, randomized order, a length penalty.
• Set up the regression gate: a set that runs automatically on every change and a written threshold; below the threshold nothing goes live.
• Sample in production and feed errors back into the set. The golden set is a living document, not a file written once.

A chatbot without a test set is a chatbot believed to work well; a chatbot with one is a chatbot known to work well. The gap between the two is 50 questions in the first week and half a day of labeling. If your chatbot is live and you still have no golden set, your last ten support tickets are a starting point for this afternoon. And if the labeling rubric is where you get stuck, the first 40 examples we scored on a similar project make a usable template.