AGENTIC LEGAL DOCUMENT REVIEW
Designing the agent-to-lawyer as a self-directed concept build. Killed three features that were costing trust without earning it.
Industry
Legal Tech
Surface
Document review
Pattern
4-stage handoff
- Type
- Concept build, self-directed
- Scope
- Product design, build, writing
- Status
- Working prototype
TL;DR
I built the agent-to-lawyer handoff as a concept. The problem I set myself: trust collapses when you ask lawyers to grade AI outputs instead of designing for the cognitive shift between AI work and human judgment.
Try it
Stage three, in your hands.
Three flagged clauses from a fictional matter. Notice what the interface refuses to let you do: accept a finding whose evidence you haven't looked at.
Lumens· Document review
Meridian & Northgate master agreement
3 clauses awaiting review
Three flagged clauses
Vendor's aggregate liability arising under this Section 8 shall not exceed the fees paid or payable by Customer in the twelve (12) months preceding the event giving rise to the claim.
The cap re-introduces the vendor's template language and silently covers data-breach claims. The negotiated term sheet carves data breach out of any cap. This clause conflicts with that agreement.
Review minutes vs. findings, by agent confidence
Findings pile up where confidence is high. Minutes should pile up where it is low. The design's job is to make that inversion the path of least resistance.
The AI and the lawyer are not doing the same job. The AI is doing fast, comprehensive, lower-confidence retrieval and synthesis. The lawyer is doing slow, narrow, high-confidence judgment.
A product that gives them the same job and asks the lawyer to second-guess the AI. That is why trust fails. The design has to be for the handoff between two cognitive modes, not for the overlap.
The reframe that scoped the build
Where the seams are.
The demo runs on written data behind real gates. The chat is where a live model call happens. Attention is the scarce resource underneath all of it: the confidence score is not decoration on a card, it is the routing signal that decides where a lawyer's minutes go.
What runs, and what was written
One box is written by hand. The rest of the demo's chain is real logic, and the site's only live model call sits in the lane below.
One box is written by hand. The rest of the demo's chain is real logic, and the site's only live model call sits in the lane below.
The decision trail
Three states of the review surface.
First pass
Accept stays disabled until every cited source has been opened.
- The verification gate went in as a working demo rather than a description of one.
- Two panes only. A flagged-clause queue beside the clause text and the agent's read.
Mid iteration
Confidence becomes a coded signal, on a color world that belongs to the product rather than the site.
- The window became a credible app. Traffic lights, centered identity, a matter toolbar.
- Confidence stopped being a ghost numeral and became a dot, a tinted meter, and a word.
Shipped state
Clause and finding each get their own titled panel, and the review opens to full size.
- Titled document and finding panels replaced the floating italic labels.
- The full view opens the way a real window would, over its own desktop.
Process artifact
One job per stage.
One job per stage, so the pattern never asks the lawyer to do two things at once. The product's record of truth is the lawyer's signature, never the model's output.
Retrieval had to surface its own scoring and the draft state its own confidence, so the design and the retrieval logic were built together, not handed off.
Retrieves
User sees a soft progress state with documents being scanned in real time.
Proposes
Agent surfaces a draft answer with citations linked inline. Distinctive draft treatment so the user knows it has not yet been reviewed.
Reviews
User reads the draft, clicks any citation to verify, and either accepts, edits, or rejects with a one-line reason.
Publishes
Reviewed answer becomes canonical. Record of truth is the human signature.
Decisions made
The calls that shaped the build.
Every project has a handful of decisions that determined the outcome. These are mine, with the trade-offs I accepted.
Killed the multi-mode review toggle. One review surface, one cognitive mode at a time.
A Quick/Detailed toggle is a confidence signal disguised as a UI setting. Wrong layer for a trust problem.
A feature teams ask for is gone. Workarounds that recreate it would be the first signal I read.
The confidence threshold slider is gone. Draft and reviewed are explicit states now.
Confidence as a continuous variable makes sense to the engineers. To the lawyer it is decision fatigue: set the threshold where, on what evidence?
The tune-it-to-your-firm story is gone. Worth watching whether anyone asks for it once the states are explicit.
No batch approve. Publishing an answer takes a review, one at a time.
The easiest feature to request and the easiest one to abuse. It lets a reviewer approve without reading.
Throughput looks slower on paper. If the friction just moves elsewhere, the design lost.
Every rejection carries a one-line reason. It trains the model and it gives the lawyer agency.
Two functions in one decision: the training loop gets structured data, and the lawyer is teaching the system rather than fighting it.
Every rejection costs a few seconds. The completion rate on that field says whether the cost was too high.
Exhibits
Four artifacts that carry the argument.
Exhibit 01 · The handoff pattern
Exhibit 02 · The reframe, on one page
Internal · re-scope memo · week 1
“Stop designing a place to read AI answers. Design the moment a lawyer takes responsibility for one. Everything upstream of that moment is the agent's problem; everything downstream is the firm's record.”
The paragraph that killed three features and set the four-stage scope.
Exhibit 03 · A draft must look like one
Draft · unreviewed0.83 moderate
The non-solicitation clause survives termination for 18 months under section 11.2…
Reviewed · signed E. Okafor
The non-solicitation clause survives termination for 18 months under section 11.2, amended per the March rider.
Exhibit 04 · A rejection teaches the model
- Reject
One click, but never the last one.
- Wrong survival period…
One required line of why.
- Routed to eval set
The reason becomes training signal.
The problem
Where it actually broke.
A legal-AI product with a strong model and a large corpus, technically impressive and commercially flat. Lawyers got accurate, cited answers. They didn't use them.
01Reading
Skim, pattern-match. Low-stakes attention.
02Reviewing
Verify, weigh, second-guess. Senior judgment.
03Deciding
Sign off, attach reputation.
Every AI output landed in the same visual treatment as a finished answer, so every output felt like a homework assignment that needed to be graded.
What I'd measure
There are no . Here is the scorecard.
If Lumens were live, this is the scorecard I would put my name to. If those numbers did not move, the pattern is wrong and I would want to know.
- Daily active use of the review surface against the pre-redesign baseline.
- The share of rejections that carry a written reason.
- Time from proposal to published answer.
- A failure-mode taxonomy for the draft state, because a draft that reads as finished is this design's most dangerous defect.
Stack used
Tools that did the work.
What I'd do differently
In .
- Instrument from day oneBuild the rejection-reason capture into Stage 3 from the start, not as a later add-on.
- Eval before flowDesign the eval for the draft-state failure mode before building the review flow, so that what 'reads as finished but isn't' shapes the visual language instead of following it.
More work