RAG vs MCP vs Fine-Tuning: Teach AI Your Firm's Work

By Jude Lee · · Custom

Small-firm attorneys and a practice manager reviewing case files and a laptop in a law office conference room

Ask a dozen partners what they want from AI and you get the same sentence back: “I want it to know our work.” Not generic legal writing — our motion templates, our fee agreements, our judge, our client’s file.

That is a solvable engineering problem, and the industry has converged on three distinct solutions with confusingly similar marketing. Legal AI vendors have been attracting large funding rounds, and the trade press regularly argues that firms should develop in-house capability rather than rent all of it. Set the noise aside: every product in that market is selling you one or more of these three techniques. Knowing which is which is how you avoid paying for the wrong one.

The three ways an AI gets your firm’s knowledge

Retrieval, usually called RAG (retrieval-augmented generation). You index a body of documents — your brief bank, your form file, closed-matter work product — into a searchable store. When someone asks a question, the system finds the most relevant passages and hands them to the model along with the question. The model answers from those passages, and can quote them. This is how an assistant says “we argued this in Whitfield, here is the paragraph.” It is good at breadth over a static corpus and it makes citation possible. It breaks when the index is stale, when the retrieval step grabs the wrong five documents, and — most dangerously — when the index flattens your access controls, so a walled matter or privileged memo surfaces to someone who should never see it.

A live connection, increasingly via MCP (the Model Context Protocol). MCP is an open standard — originally published by Anthropic and since adopted by other assistants and tool vendors — for giving an AI governed access to real systems and actions: read this matter, list these deadlines, create this document, log this time entry. The difference from retrieval is that MCP is live and transactional. It queries the system of record at the moment of asking, respects the permissions of the user it’s acting for, and can write back. Maturity is uneven, though: as of 2026, read access is far more common than write-back, and how finely you can scope permissions depends on both the assistant you use and what your practice-management vendor’s API actually exposes. That unevenness, not caution for its own sake, is why read-only is the right starting posture. We covered the mechanics in connecting an assistant to your own matter data.

Fine-tuning. You take a base model and train it further on a few hundred or few thousand of your own examples — your demand letters, your engagement letters — so its default output sounds like your firm. Fine-tuning changes the model’s behavior and style. It does not reliably teach it facts, it does not update when a case gets decided, and it has no concept of who is allowed to see what. A fine-tuned model that has absorbed client work product is also a confidentiality question you should walk through deliberately.

Retrieval over your document store
Best for: “Have we briefed this before?”, form selection, precedent search, onboarding a new associate to the firm’s house arguments.

Fails at: current matter status, anything transactional, and permission-sensitive material — a flattened index can leak privileged or ethically walled work product to the wrong querier.

Effort: moderate. The hard part is corpus hygiene — deciding which closed matters are good law and good work.
Live MCP connection to your systems
Best for: matter status, deadlines, contacts, conflicts data, billing, and taking action — drafting into the DMS, opening a matter, logging time.

Fails at: semantic search across unstructured work product; it’s only as good as the API behind it.

Effort: moderate to high, and it needs a named owner — see the staffing note below.

What a real build looks like in a 14-lawyer firm

Say you’re a litigation boutique and the recurring pain is first-draft opposition briefs. Here’s how the three layers stack in practice.

  1. Start with a written skill, not a model change

    A skill is a packaged, reusable instruction set: our brief structure, our citation conventions, where the standard-of-review paragraph goes, what we never say. This is plain text, versioned, reviewable by a partner. In our view it solves most of what people think fine-tuning is for — see how firms build skills for repeatable review.

  2. Add retrieval over a curated brief bank

    Not “all of NetDocuments.” Pick the filings a senior associate would actually hand someone as a model, and exclude anything under a wall. Curation beats volume; a corpus full of superseded arguments produces confidently wrong drafts.

  3. Connect the live systems over MCP, read-only first

    Matter metadata, parties, deadlines, the documents on this file. Read-only until the logs are boring. Then, and only then, consider write access for narrow actions like saving a draft to the correct matter folder.

  4. Name who actually does the work

    A 14-lawyer firm rarely has an engineer on staff, and this is the real gate. Realistically it’s one of three: your practice-management vendor’s own integration or marketplace connector, your managed IT provider if they have genuine API experience, or an outside integrator for the custom pieces. Either way, someone inside the firm — usually the firm administrator or a tech-minded partner — has to own permissions, access reviews, and the decision log. If no one will own it, stop at the skill layer.

  5. Decide on fine-tuning last — and probably decide against it

    If after the earlier steps the output still doesn’t sound like you, revisit the skill. Fine-tuning adds cost, a retraining cadence, and a data-governance conversation. It is rarely what a small firm is actually missing.

Firms think they have a model problem. They almost always have a retrieval problem, a permissions problem, or an unwritten-standards problem.

Where firms are actually running AI today

The honest answer, as of early 2026, is four overlapping layers. First, AI features inside the practice-management platform you already pay for — Clio, MyCase, Smokeball and peers have all shipped assistant features, but scope and availability differ by product, plan and region, so check the vendor’s own documentation rather than assuming parity. Second, research platforms: Westlaw, Lexis+ AI, Bloomberg Law’s AI tools, which bring their own licensed corpus and citator. Third, legal-specific drafting and review products such as CoCounsel, Harvey, Spellbook and Legora. Fourth, a general assistant — Claude, ChatGPT, Copilot — connected to firm systems.

The first three are mostly retrieval plus tuned prompting over someone else’s corpus, or yours inside their container. The fourth is where MCP matters, because the knowledge is yours and the connection is governed by you. If you’re weighing a document-level tool against a connected setup, our comparison of NotebookLM, Claude Projects and MCP for case files walks the same tradeoff at smaller scale.

Modeling the return without inventing numbers

We have no benchmark data on what this saves, and you should distrust anyone who quotes you one. Model it yourself:

(hours per task now − hours per task after) × tasks per month × realization-adjusted hourly rate

Then subtract build and license cost, and subtract review time — because a draft a partner has to re-verify line by line hasn’t saved a full hour. Plug in your own figures.

And be precise about what the recovered hours are worth. Automation doesn’t conjure revenue; it shifts time out of non-billable drafting and file-hunting and into billable work or business development, and it reduces billing leakage. That reallocation only counts if the hours actually land somewhere productive — so decide in advance where they go, and track whether they did.

hrs × rate
The only ROI formula that survives contact with a managing partner
Illustrative framework — use your firm's actual figures
200–400
Example brief-bank size to start with — curated, not exhaustive
Illustrative starting point, adapt to your practice

What has to stay human

Everything cited, everything filed, everything communicated to a client. ABA Formal Opinion 512 (2024) addresses generative AI use and reinforces the duties of competence, confidentiality, and reasonable fees; Model Rule 1.1 Comment 8 covers technology competence. Your state bar may have issued its own guidance with different specifics — check the primary source rather than a summary, and confirm anything with real compliance stakes with a qualified professional.

The practical conclusion is unglamorous: write the skill, curate the corpus, connect the systems with narrow permissions and a named owner, and leave fine-tuning on the shelf until you have a specific, measured reason to take it down.

Where is your firm losing billable hours?

Get a free automation audit: we map your intake-to-invoice workflow and show you exactly what's worth automating — before you spend a dollar.

Get a free automation audit

Related guides