Evaluating LLMs and agents

How to tell whether a gen AI system is good enough and stays that way: building evaluation datasets, computation-based vs model-based (LLM-as-a-judge) metrics, judge biases and how to control them, pointwise vs pairwise and rubric-based evaluation, Agent Platform Evals, agent trajectory metrics, safety evaluation, online monitoring, and statistics for comparing versions. Includes a win-rate confidence calculator.

1. Why LLM evaluation is different

Classic models have one right label per input. Generative outputs are open-ended: many answers can be right, wording varies run to run, and quality has several dimensions (correct, grounded, safe, helpful, on-format). So evaluation combines several kinds of metric over a fixed, versioned dataset, and runs on every change.

2. Building the evaluation dataset

• Representative: sample real user questions (after removing PII), covering intents, languages and difficulty.
• Edge and adversarial cases: out-of-scope questions, ambiguous requests, prompt-injection attempts, questions the documents don't answer.
• References where possible: expected answers, expected source documents (for RAG), expected tool calls (for agents).
• Size: dozens for quick iteration, hundreds for release decisions; more for small differences (see the win-rate calculator).
• Versioned and frozen: store in Cloud Storage or BigQuery; never tune prompts on the final test split.
• Add every production failure you find to the dataset (it becomes a regression test).

3. Computation-based vs model-based metrics

4. Pointwise, pairwise and rubrics

5. LLM-as-a-judge: biases and how to control them

Validating the judge

1. Have humans label a sample (for example 100–200 items) with the same rubric.
2. Measure agreement between judge and humans: accuracy, and Cohen's kappa (agreement corrected for chance), or correlation for scores.
3. Iterate on the judge prompt and rubric until agreement is acceptable; recheck when the judge model changes.

Kappa interpretation (rough): below 0.4 weak, 0.4–0.6 moderate, 0.6–0.8 substantial, above 0.8 strong. Use the calculator below.

6. Agent Platform Evals

Agent Platform Evals (formerly the Gen AI evaluation service) runs evaluations of models, prompts, RAG systems and agents.

# Sketch with the Agent Platform SDK; module and metric names vary by release.
import pandas as pd
import vertexai
from vertexai.evaluation import EvalTask, PointwiseMetric, MetricPromptTemplateExamples

vertexai.init(project="PROJECT_ID", location="us-central1")
data = pd.DataFrame({"prompt": prompts, "response": responses, "reference": references})
task = EvalTask(dataset=data,
                metrics=["exact_match", "rouge_l_sum",
                         MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS],
                experiment="support-bot-eval")
result = task.evaluate()
print(result.summary_metrics)

7. Evaluating agents: trajectories and tool use

For agents, the final answer isn't enough: the trajectory (sequence of tool calls) shows whether the agent got there safely and efficiently.

8. Safety evaluation and red-teaming

• Red-teaming: deliberately try jailbreaks, prompt injection (direct and through documents or tools), harmful requests, data exfiltration and PII leakage before launch.
• Safety metrics: model-based safety scores, policy-violation rates, refusal rates on harmful prompts, over-refusal rates on benign prompts.
• Track both under-blocking (harmful content gets through) and over-blocking (legitimate users refused).
• Fairness: compare quality and refusal rates across languages, dialects and user groups.
• Controls under test: safety filters, Model Armor templates, system instructions, tool permissions.

9. Online evaluation and monitoring

10. Regression testing in CI/CD

1. Store prompts, model version, retrieval settings and tools in version control.
2. On every change, a pipeline (Cloud Build or a Pipelines run) generates responses for the frozen evaluation set.
3. Compute computation-based checks (schema, exact match, tool calls) and model-based metrics; compare with the current production baseline.
4. Gate: block the release if key metrics drop beyond a tolerance or safety checks fail.
5. Log the run to Experiments for history; pin model versions so upgrades are deliberate and evaluated.

11. Win-rate and judge-agreement calculator

Pairwise win rate with a 95% Wilson interval (ties excluded), and Cohen's kappa between the judge and human labels.

12. Lab: evaluate two prompt versions and an agent

1. Build a 50-row dataset from realistic questions with references (and expected sources for RAG).
2. Run prompt A and prompt B. Compute exact match or ROUGE where references fit, and a pointwise groundedness or instruction-following metric.
3. Run a pairwise comparison of A vs B; then swap the response order and rerun. How many verdicts flip (position bias)?
4. Label 40 rows yourself; compute judge–human kappa with the calculator.
5. For an ADK agent, write expected tool trajectories for 10 tasks; compute exact, in-order and any-order match, precision and recall.
6. Wire the evaluation into a Cloud Build step that fails when groundedness drops by more than a set tolerance.

13. "Which evaluation method?" chooser

Pick an item to see what it means, why, and the tempting wrong choice.

14. Sample outputs

What the tools return. Expand an item to see a typical result and what to read in it.

metric                       candidate (grounded)   baseline (ungrounded)
groundedness (pointwise, 1-5)        4.6                    3.1
pairwise_quality win rate            0.71                   0.22   (ties 0.07)
rouge_l                              0.48                   0.41
avg latency (s)                      1.9                    1.2

15. Scenarios

Exam-style situations: how to diagnose each one, the likely cause and the fix.

16. Searchable reference

17. Flashcards (click to flip)

18. Common gotchas in one list

19. Practice quiz (22 questions)