How to Test a Legal AI Agent: Spot Checks vs Golden Sets

By Jude Lee · · Workflow

Lawyers and a paralegal comparing printed documents against AI-generated output on a laptop in a law firm conference room

Why accuracy testing suddenly became the bottleneck

The pitch for agentic AI is that it takes multi-step actions: read the intake email, run the conflict check, open the matter, draft the engagement letter. The hard part is that a seven-step agent can fail silently at step three and still produce a confident, well-formatted result at step seven. A chatbot that gives a bad answer is obvious. An agent that filed a document to the wrong matter is not.

The skepticism we hear from litigators about agents is rarely about raw model capability. It is that nobody defined what “correct” means for the task, so nobody can tell whether the agent is drifting — or by how much.

The professional obligation is already clear enough. ABA Formal Opinion 512, issued in July 2024, addresses generative AI and the duties of competence, confidentiality, and supervision — including the expectation that lawyers verify outputs rather than take them on faith. Several state bars have issued their own guidance; check yours directly, because the specifics vary. And the consequences of skipping verification are documented in the record: in Mata v. Avianca (S.D.N.Y. 2023), the court sanctioned attorneys who filed a brief containing fabricated case citations.

Spot checks: cheap, fast, and systematically misleading

A spot check is what nearly everyone does by default: run the agent, read a few outputs, decide it looks fine. It is genuinely useful — it catches catastrophic breakage, format problems, and tone issues in minutes, with no setup.

Its weakness is selection. You tend to check the first few, which are the ones you were paying attention to, on the documents you happened to have handy. A deposition summary agent that handles clean transcripts beautifully and mangles ones with two speakers talking over each other will pass a spot check every time until the day it doesn’t.

Spot checks are the right method for low-stakes, low-volume, reversible tasks: summarizing an email thread for your own reading, first-draft brainstorming, triaging your own inbox. They are the wrong method for anything that writes into your practice management system or goes out the door.

Golden sets: the method most firms skip and later wish they hadn’t

A golden set is a fixed collection of real (de-identified) documents or inputs for which you have recorded the correct answer. Thirty commercial leases with the rent commencement date, assignment clause, and termination rights already pulled by a human. Fifty intake emails already labeled as qualified, unqualified, or conflict-risk. Twenty discovery requests with the objections your firm actually asserts.

You run the agent over the whole set, score it against the known answers, and keep the score. Then — and this is the entire point — you run it again every time something changes: a new model version, an edited prompt, a revised skill definition, a vendor’s silent backend update.

30–50
Items in a starter golden set — a starting heuristic to size against your own matter volume and variety, not a measured benchmark

A few rules of thumb we’d stand behind, offered as opinion rather than findings:

That last point matters more than it sounds. A 40-item set produces a noisy score. If accuracy moves from 36/40 to 34/40, that two-item swing is not, on its own, evidence of a regression — it could be the same model making a borderline call differently. Open the two items that flipped. If they are edge cases you already knew were ambiguous, log it and move on. If a field that was previously rock-solid started failing, you have a real signal. Diagnose by item, not by percentage.

Building a set is less work than it sounds, because you already have the answers — they are sitting in closed matters.

  1. Pick one task with a crisp right answer

    Field extraction, classification, and routing are ideal. Open-ended drafting is harder to score and should wait.

  2. Pull 30–50 closed-matter examples, weighted toward the ugly ones

    Include the scanned fax, the handwritten amendment, the lease with a typo in the term. Clean documents teach you nothing.

  3. Record the correct answer in a plain spreadsheet

    One row per document, one column per field or label. A paralegal who already does this work is the right person to populate it.

  4. Decide in advance how you resolve disagreement

    Lawyers will not always agree on the right answer. Have a second reviewer check a slice of the annotations, and when the two disagree, either escalate to the supervising attorney or write down a documented house rule (“assignment clauses with consent-not-unreasonably-withheld language are coded as permissive”). Otherwise your ground truth is one person’s judgment wearing a lab coat.

  5. Define scoring before you run anything

    Exact match for dates and dollar amounts. For classification, track false positives and false negatives separately — missing a conflict is not the same error as flagging a phantom one.

  6. Run, score, and write down the number with the date and model version

    If you can’t reconstruct what configuration produced a score, the score is decoration.

  7. Set a floor that triggers human review

    Below your threshold, the task goes back to manual or to full human verification until it’s fixed.

  8. Refresh the set on a schedule

    Golden sets go stale. Retire items whose clause language or firm practice has changed, add new examples whenever a new failure mode shows up in production, and review the whole set at least annually — or sooner if it stops looking like the work actually coming in the door.

Dual-run shadow mode: the only way to know if it’s actually better

In a dual run, the agent and your existing process both handle live work for a defined window. Nothing the agent produces goes out the door; a human compares. You learn three things a golden set can’t tell you: how the agent handles the real distribution of incoming work, how long verification actually takes, and whether the humans trust it enough to use it.

To make “net time saved” more than a feeling, instrument it. Have each reviewer log two things per item: an agree/disagree flag, and minutes-to-verify. Compare that median verification time against your baseline time to do the task from scratch. If verifying takes nearly as long as doing, the agent isn’t saving you anything — a finding worth discovering in week two rather than month six. Trust shows up in the same log: if disagreement clusters with one reviewer, that’s a training or interface problem, not a model problem.

Golden set
Fixed, repeatable, cheap to re-run. Catches regressions over time. Built once, maintained with periodic refreshes. Blind to inputs you never thought to include, and says nothing about whether the workflow is faster end to end.
Dual run
Reflects real incoming work and real review burden. Measures net time saved, not just correctness — if you log verification minutes. Expensive, since you’re paying for the work twice, and you can’t re-run last month’s conditions.

A reasonable sequence: golden set to decide whether the agent is plausible, dual run for two to four weeks to decide whether it’s worth deploying, golden set again, permanently, to confirm it stays that way.

A benchmark score tells you the agent can do the job. Only a parallel run tells you whether your firm gets anything back from it.

Scoring different kinds of agent work

Not every task scores the same way, and using the wrong yardstick produces confident nonsense.

Extraction (lease terms, medical record dates, policy limits) scores on field-level exact match. Report per-field, not overall — an agent that looks strong overall may be weak on the one field that matters.

Classification and triage scores on false positives versus false negatives, kept separate. In discovery triage, over-inclusion costs review hours; under-inclusion costs you a document you needed. Those are different problems with different fixes.

Drafting needs a human rubric: did it use the firm’s clause, cite only real authority, follow the local format? Citation accuracy deserves its own check — see the options for automated citation verification.

Action-taking agents need a fourth category nobody scores enough: unsafe actions. Did it write to the right matter? Email the right party? Trigger anything irreversible? One unsafe action in a hundred runs is a different risk profile than one bad summary in a hundred.

When testing this hard isn’t worth it

If a task is deterministic — same input, same output, no judgment — you probably want a rule-based automation, not an agent, and the testing problem mostly disappears. If a task runs twice a year, the setup cost of a golden set exceeds the value. If output is reviewed line by line by a lawyer anyway, a lightweight spot-check cadence plus a clear oversight model may be sufficient.

The honest economics: building a golden set costs a defined number of paralegal hours, once, plus a smaller re-run cost per change and a periodic refresh. Weigh that against your own estimate of error cost — rework hours at your blended rate, a missed deadline, a client relationship, a malpractice exposure. We can’t tell you the breakeven; we don’t have your numbers. But for any workflow where an agent touches client-facing output more than a few times a week, our view is that the spreadsheet pays for itself the first time a model update quietly changes behavior and you catch it in an afternoon instead of a filing.

Where is your firm losing billable hours?

Get a free automation audit: we map your intake-to-invoice workflow and show you exactly what's worth automating — before you spend a dollar.

Get a free automation audit

Related guides