Supervising AI Agents: 3 Oversight Models for Law Firms

By Jude Lee · · Workflow

Small-firm lawyers and a paralegal reviewing AI-drafted work together at a conference table

Why “review the output” stopped being an oversight plan

Generative AI produces text you read before anything happens. Agentic AI plans a multi-step task and takes actions in your systems — the distinction Thomson Reuters walks through in its comparison of agentic and generative AI. We covered the same split from a practice angle in agentic AI vs generative AI in law firms.

That difference changes supervision entirely. A bad paragraph in a draft is caught by reading. A bad action — a matter opened under the wrong client, a document saved to the wrong matter folder, an email sent to opposing counsel — has already happened by the time you read about it. Commentary like TechRadar’s piece on what happens when AI agents go rogue makes the obvious point that accountability doesn’t transfer to the software. In a law firm it lands on the supervising lawyer.

Oversight isn’t a review habit. It’s a permission model. If the agent can’t take the action, you don’t need the discipline to catch it.

How AI shows up in small and mid-size firms

A useful way to sort firm AI use — offered as a framing, not as survey data — is four buckets: drafting (correspondence, discovery responses, demand letters), summarizing (depositions, medical records, long email threads), searching (research, internal precedent, matter files), and routing (intake triage, calendaring, email sorting). The first three are generative. The fourth is where agents start touching systems of record.

My view: firms get value fastest from summarizing and drafting, because a human is already the last step. Agentic workflows can go further, but only after someone decides which steps a machine may complete unattended.

Sort by reversibility and commitment

Lawyers use “80/20” loosely — a small share of tasks drives most of the risk. Apply it to oversight rather than revenue. Sort every AI-assisted step by two questions:

  1. Is the action reversible? Draft in a folder: yes. Filed with the court, sent to a client, deleted from the DMS: no.
  2. Does it commit the firm? Legal conclusions, deadlines, fee statements, and anything a client will rely on commit you. Internal summaries mostly don’t.

Everything irreversible or firm-committing goes in the tight-oversight bucket. Everything else can run looser. That sorting exercise — not a vendor’s feature list — should determine your model.

Three oversight models, compared

1. Draft-only (copilot). The AI produces work product; a human moves it. No system writes, no sends. Cheapest to govern, slowest to scale, and the right default for anything client-facing or substantive. Most off-the-shelf legal AI features sit here by design.

2. Approval gates (human-in-the-loop). The agent runs a multi-step workflow but pauses at defined checkpoints and waits for a named human to approve. Steps before the gate are read-only or internal; steps after it carry consequences. This is the workhorse model for small firms.

3. Supervised autonomy (act, then audit). The agent completes a narrow workflow end to end, logs everything, and a human reviews samples plus all exceptions. Appropriate only for low-stakes, high-volume, reversible actions — tagging emails to matters, filing signed documents into a known folder structure, populating internal fields.

Approval gates
Best for: intake and matter opening, document assembly, discovery response drafting, anything with a filing deadline.
Cost: a real person’s attention in the loop, every time. Throughput is capped by reviewer availability.
Failure mode: rubber-stamping. Gates that always get approved are theater.
Supervised autonomy
Best for: filing, tagging, indexing, internal summarization, status updates that stay inside the firm.
Cost: you must build logging, exception flags, and a sampling routine — or you have no oversight at all.
Failure mode: silent drift. If nothing samples the output, mis-tagging can run for weeks before anyone notices.

What the ethics rules actually expect

The ABA’s Formal Opinion 512 (July 2024) addresses generative AI tools and touches competence, confidentiality, communication with clients, fees, and — importantly here — supervisory duties under Model Rules 5.1 and 5.3. Read it directly rather than through a vendor summary. Several state bars have issued their own guidance, including the State Bar of California’s practical guidance on generative AI and the Florida Bar’s Ethics Opinion 24-1; emphases differ by jurisdiction, so confirm your state’s position with the primary source and, where a specific arrangement has real stakes (client consent, fee treatment, a vendor’s data handling), with qualified ethics counsel.

Building the gate into the tools, not the instructions

Most “AI policy” documents miss the practical point: telling an agent to “ask before filing” is a request. Not exposing a filing tool is a control.

There are several ways to enforce that. Native vendor integrations, a plain API script with scoped API keys, or an RPA/no-code workflow tool can all enforce the same read/write split. MCP — an open protocol for giving an AI governed access to tools and data — is worth the setup mainly when you want one assistant working across several systems under one permission model. A Claude-to-Clio MCP setup can expose search_matters and get_document while simply not exposing create_matter or send_email. If you build a custom MCP server over your firm’s data, you also control scoping, logging, and staged writes.

A worked example: the intake agent with gates

  1. Capture and enrich (autonomous)

    Agent pulls the web form or call transcript, normalizes names, identifies parties and case type, and writes a structured summary to a review queue. Reversible, internal — no gate needed.

  2. Run the conflict search (autonomous, read-only)

    Agent queries your conflicts data and returns matches with underlying records attached. It surfaces evidence; it clears nothing. See our walkthrough on automating conflict-of-interest checks.

  3. GATE 1 — a lawyer clears the conflict

    A judgment call with reputational stakes. Named approver, recorded decision.

  4. Open the matter and generate the engagement letter (autonomous, draft state)

    Matter created in pending status; engagement letter assembled from your template with fee terms populated but unsent.

  5. GATE 2 — lawyer reviews fees and scope, then sends

    Fee terms and scope commit the firm. Human sends.

  6. Post-send housekeeping (autonomous, audited)

    Calendar the follow-up, file the signed copy, update the CRM. Sampled weekly against the log.

Two gates. Everything else runs. That’s the shape most small-firm agentic workflows should take.

Where agents break — and what stays off-limits

The recurring weaknesses are worth stating plainly: confident fabrication (including citations), silent failure on documents it can’t parse — bad scans, handwriting, non-standard forms — brittleness when a system’s UI or API changes, and review fatigue, where oversight decays into clicking approve. Add change management: an agent nobody trusts gets bypassed, and an agent everybody trusts blindly is worse.

Some steps aren’t gate-able at all — they’re the lawyer’s own act, not a task to supervise. Signing and filing a paper under a lawyer’s signature; deciding whether a conflict is waivable and counseling the client on that waiver; recommending acceptance or rejection of a settlement offer. An agent can assemble the packet and surface the evidence. It cannot be the one exercising the judgment, no matter how many approvals sit on top.

A rule-based automation is often the smarter call. If the task has no judgment in it — copy field A to field B, calendar 30 days from service — use a workflow rule or a no-code automation. Cheaper, deterministic, auditable without a sampling program.

Costing oversight without inventing numbers

Any ROI case that ignores reviewer time is fiction. Model it yourself with your own measured figures — this is an illustrative formula, not a benchmark:

  1. Gross time recovered = minutes saved per matter × matters per month. Measure a baseline before you start.
  2. Subtract oversight cost = review minutes per gate × number of gates × matters per month.
  3. Convert to money = net hours × your realization-adjusted hourly rate — but only for hours someone actually refills with client work or intake follow-up.

If the net is negative, you have a workflow that needs fewer gates or an automation that shouldn’t be an agent.

Choosing a model per workflow

There is no single best AI program for a law firm, and firms that shop that way tend to churn tools. What travels between workflows is the oversight decision. Start every AI project by asking which of the three models applies: draft-only work is usually well served by off-the-shelf features already in your practice-management or research platform; approval-gated multi-step workflows are where custom agents and a scoped tool layer start earning their keep; supervised autonomy belongs to the narrow, boring, reversible tasks you can verify by sampling.

Write the model down next to each workflow, name the approver, and review the logs monthly. That document is worth more than any tool selection.

Where is your firm losing billable hours?

Get a free automation audit: we map your intake-to-invoice workflow and show you exactly what's worth automating — before you spend a dollar.

Get a free automation audit

Related guides