AI LEGAL DOCUMENT REVIEW
Lumens AI
Designing the handoff between agent analysis and lawyer judgment.
Killed three features that let a lawyer skip the reading that makes an acceptance mean something. A finding cannot be accepted until its evidence has been opened, and an override keeps its reason on the record.
TL;DR
The agent can find and cite. The lawyer still has to decide. Lumens is built around that handoff.
The turning point
The features that let a lawyer skip the reading were cut.
I worked with the founding team on Lumens as product design lead, from the first brief to the working build.
The first cut gave the agent and the lawyer the same job: find the risky clause, grade it, then ask the lawyer to second-guess the grade.
- Clause 9.1
- Clause 9.3
- Clause 11
- Clause 12
Batch approve skipped the reading
A reviewer could accept a whole set of findings without reading one.
Agent
High risk
- Liability capHigh
- Carve-outsHigh
- IndemnificationHigh
- AssignmentHigh
Lawyer
?
- ?
- ?
- ?
- ?
Two parties, one job
The first cut had the agent find and grade the risky clause, then asked the lawyer to second-guess the grade.
Clause 9.1
Liability cap
- FindingAccepted
- SourceNot opened
An acceptance meant nothing
A finding could be accepted while its source sat unopened.
Review mode
QuickDetailedConfidence
LowHighA toggle to skip the reading
The Quick/Detailed toggle and the confidence slider each let a lawyer skip the reading.
Try it
Now make the call.
Three flagged clauses from a fictional matter. Notice what the interface refuses to let you do: accept a finding whose evidence you haven't looked at. The clause data is authored. Source tracking, review routing and approval gates run as code.
Ashworth Health, Matter 2026-0412
Master services agreement
1 Definitions
“Confidential Information” means non-public information disclosed by either party that is marked confidential or that a reasonable person would understand to be confidential from its nature. “Services” means the contract-review software and implementation work described in Schedule A. “Data Security Incident” means unauthorized access to, or disclosure of, Client data processed under this Agreement.
2 Services
Vendor shall provide the Services in accordance with the specifications in Schedule A and the service levels in Schedule B. Vendor shall notify Client promptly of any change that would materially reduce the Services' functionality.
3 Fees
Client shall pay the fees set out in Schedule C on a monthly basis, net 30 from the invoice date. Fees are exclusive of applicable taxes, which Client shall bear in addition to the amounts stated, and unpaid balances accrue interest at 1.5% per month.
4 Term
This Agreement commences on the Effective Date and continues for 36 months, renewing automatically for successive 12-month periods unless either party gives 90 days' written notice of non-renewal.
5 Confidentiality
Each party shall protect the other's Confidential Information using at least the same degree of care it applies to its own, and in no case less than reasonable care. This obligation survives termination for five years.
6 Intellectual property
Vendor retains all rights in the Services and underlying software. Client owns its own data and grants Vendor a limited license to process that data solely to provide the Services.
7 Data Security
Each party shall maintain reasonable safeguards for data handled under this Agreement.
9 Limitation of Liability
The limits, exclusions, and allocation of risk in this Section govern the parties' claims.
9.1 Cap
Except as provided in Section 9.3, neither party's aggregate liability under this Agreement, including claims arising under Section 7 (Data Security), shall exceed the fees paid or payable in the twelve (12) six (6) months preceding the claim.
9.2 Exclusions
These limits do not apply to indemnification under Section 11, or to gross negligence.
9.3 Carve-outs
Claims arising under Section 7 (Data Security) are subject to the separate cap set out in the Term Sheet.
9.4 Survival
The limitations and exclusions in this Section 9 survive termination or expiry of this Agreement.
9.5 Aggregate application
Where more than one claim arises from a single course of conduct, those claims shall be treated as one claim for the purpose of the cap.
11 Indemnification
Each party's indemnification obligations under Section 11 remain outside the cap in Section 9.1.
12 Assignment
Neither party may assign this Agreement without the prior written consent of the other party, subject to the change-of-control language in Section 12.3.
Why the evidence stays in view
Legal research benchmarks provide context for this review approach. They do not evaluate Lumens or its contract-review workflow.
How three legal AI tools answered
A 2024 Stanford study. Each mark is one percentage point of a tool’s responses. Lumens was not part of it.
Study notes
The three tools answered the same preregistered set of legal research questions, March to May 2024. Correct and grounded means right, and citing a source that supports it. Hallucinated means it states something false or cites a source that does not support it. Incomplete means it does not answer or does not cite.
Each grid is 100% of a tool’s responses, in the paper’s rounded figures; a mark is a percentage point, not a single tested answer. Westlaw’s figures follow the paper’s abstract and introduction (42% correct, 33% hallucinated); its results section gives 41% correct, and Stanford HAI’s summary says more than 34% hallucinated. Ask Practical Law AI’s hallucinated share is the remainder after 19% and 62%, which the paper states as over 1 in 6.
Where it came from
The starting point is the shape review tools in this category usually take. Every matter, clause and name shown is invented for the prototype.
From a list of findings to a decision
Findings beside a document parked elsewhere, then a review that lands on the clause.
My role
I led Lumens from the brief to the working build.
My responsibility
I decided what the agent may propose, what a lawyer must see before accepting it, and what stays on the record.
I worked with the founding team on the product definition, the review surface and a working build engineering could take forward.
§ 9.3Source passage opened
Proposed finding
Awaiting lawyer decision
Edits and reasons stay on the record.
How I worked
Wrote the brief before any screen. It named the problem first: a lawyer could not tell which output to rely on.
Built the interface and the retrieval together. Retrieval had to expose its own scoring and drafts their own confidence, so the two were built side by side.
Wrote the scorecard a pilot would check. Review time should pile up where agent confidence is low. That is the first number a pilot would read.
“The AI and the lawyer are not doing the same job. The AI is doing fast, comprehensive, lower-confidence retrieval and synthesis. The lawyer is doing slow, narrow, high-confidence judgment.”
The reframe that scoped the build
Lumens / Review workflow
Evidence before acceptance.
Ashworth MSA v4 / Clause 9.1
Limitation of liability
Clause 9.1
Except as provided in Section 9.3, neither party’s aggregate liability under this Agreement, including claims arising under Section 7 (Data Security), shall exceed the fees paid or payable in the six months preceding the claim.
Clause 9.3
Claims arising under Section 7 (Data Security) are subject to the separate cap set out in the Term Sheet.
Check the exception before recording a decision.
Inspect the source
Trace a finding back to the clause and its cited evidence.
Review cited evidence
1 of 2 sources opened
Open both sources to continue.
Review before applying
Apply edit unlocks after every cited source is opened.
Decision recorded
Finding 01 · Clause 9.1
- Decision
- Override
- Reviewer
- RK · 09:42
Reason
Clause 9.3 refers to the term sheet’s uncapped position. Clarify the cross-reference rather than remove the carve-out.
Reason retained with finding.
Keep the reasoning
Keep the lawyer’s override and its reason on the record.
Where the seams are.
Confidence determines where review attention goes. The lawyer inspects the evidence and makes the decision.
neither party's aggregate liability, including claims arising under Section 7 (Data Security), shall exceed the fees paid
Clause data, written by hand
The cap now covers data-breach claims.
Score routing, runs as code
Sources read, runs as code
Approval gate, runs as code
After approval
Signed record, runs as code
Separate: site chat
One part of the card is written by hand. The rest of the demo's chain is real logic, and the site's only live model call sits in the lane below.
The decision trail
From an assertion to an inspectable finding
Reconstructed progression · Fixed fictional matter
Assertion alone
The reviewer has to locate the basis.
LumensDocument reviewAshworth MSA v4Potential conflict in the liability cap
Draft findingSection 9.1 includes data-security claims within the cap.
Evidence attached
Source references become inspectable.
LumensDocument reviewAshworth MSA v4Potential conflict in the liability cap
Draft findingSection 9.1 includes data-security claims within the cap.
MSA v4 · §9.1
“… including claims arising under Section 7 (Data Security) …”
Term sheet · §3
“Data-protection liability is uncapped.”
Supporting note
Check the exception in §9.3 before deciding.
Conflict in context
Read the operative language, exception and proposal together.
LumensDocument reviewAshworth MSA v4Current MSA
§9.1Limitation of liability
Except as provided in Section 9.3, the total aggregate liability of either party shall not exceed the fees paid or payable under this Agreement, including claims arising under Section 7 (Data Security), whether in contract, tort or otherwise.
§9.3Exception
Claims arising under Section 7 (Data Security) are subject to the separate cap set out in the Term Sheet.
Term sheet · §3
§3Liability
Data-protection liability is uncapped.
Potential inconsistency
Read §9.1 together with §9.3 and the term sheet. Confirm the intended treatment before editing.
Proposed clarification
Clarify the cross-reference in §9.1.
The interface exposes the basis of the finding; the lawyer determines its legal effect.
Illustrative reconstruction, not a dated version record.
The handoff
One job per stage.
One job per stage, so the pattern never asks the lawyer to do two things at once. The product's record of truth is the lawyer's signature, never the model's output.
Retrieval had to surface its own scoring and the draft state its own confidence, so the design and the retrieval logic were built together, not handed off.
- Ashworth MSA v4
Risk score 71 of 100
2 failed1 warning14 passed
1Retrieves
Documents are scanned and scored against the playbook.
- Draft · Not reviewed
Data liability sits outside the cap
Confidence highTerm sheet, clause 32Proposes
A draft arrives wearing its citation, marked as not yet reviewed.
Data liability sits outside the cap
AcceptRead source to enable AcceptSource reviewedRejectEditAccept3Reviews
Accept stays locked until the cited source has been read.
- Accepted
Data liability sits outside the cap
Edit applied to clause 9.1: data-security claims remain uncapped.
RK at 09:424Publishes
The accepted finding is signed. The record of truth is the signature.
Decisions made
The calls that shaped the build.
Every project has a handful of decisions that determined the outcome. These are mine, with the trade-offs I accepted.
No batch approve. Publishing an answer takes a review, one at a time.
It is the easiest feature to ask for and the easiest to abuse, because it lets a reviewer approve without reading. The Quick/Detailed toggle and the confidence slider went for the same reason: each let a lawyer skip the reading that makes an acceptance mean something.
Human-AI guidance and leading legal review products tie correction, review and verification to individual suggestions or cells.
Throughput looks slower on paper. If the friction just moves elsewhere, the design lost.
Every rejection carries a one-line reason.
A rejection with a reason is feedback instead of a dead end. It keeps what the lawyer changed, and why, on the record.
Human-AI guidance recommends collecting explicit commentary on AI output, and NIST notes that the rationale behind human overrides is useful evidence to analyze.
Every rejection costs a few seconds. The completion rate on that field says whether the cost was too high.
Exhibits
Two artifacts that carry the argument.
Exhibit 01A draft must look like one
Draft, not yet reviewedModerate confidence
The non-solicitation clause survives termination for 18 months under section 11.2…
Reviewed and signed by D. Kessler
The non-solicitation clause survives termination for 18 months under section 11.2, amended per the March rider.
Exhibit 02A rejection becomes structured signal
- Reject
One click, but never the last one.
- Wrong survival period…
One required line of why.
- Routed to eval set
The reason becomes structured signal.
The problem
The trust gap.
The premise Lumens starts from: a legal-AI product with a strong model and a large corpus can return accurate, cited answers and still go unused, because nothing tells the lawyer what to trust.
Reading
Skim, pattern-match. Low-stakes attention.
Reviewing
Verify, weigh, second-guess. Senior judgment.
Deciding
Sign off, attach reputation.
Every AI output landed in the same visual treatment as a finished answer, so every output felt like a homework assignment that needed to be graded.
Legal research providers marketed retrieval as eliminating hallucinations and promised hallucination-free citations. The first preregistered evaluation of those tools found the claims overstated.
What I'd measure
Here is the scorecard.
This is the scorecard I would put my name to. If those numbers did not move, the pattern is wrong and I would want to know.
Data liability sits outside the cap
Daily use of the review surface.
Against the pre-redesign baseline.
Draft, not yet reviewed
The non-solicitation clause survives termination for 18 months under section 11.2
Rejections that carry a reason.
The completion rate on that reason field is this design's KPI.
Time to a published answer
Time to a published answer.
From proposal to the signed finding.
The non-solicitation clause survives termination for 18 months under section 11.2, amended per the March rider.
A taxonomy of draft-state failures.
A draft that reads as finished is this design's most dangerous defect.
What I would test
The numbers below are an illustrative target a pilot would check.
Review minutes vs. findings, by agent confidence
Findings pile up where confidence is high. Minutes should pile up where it is low. The design's job is to make that inversion the path of least resistance.
What I'd do differently
In retrospect.
Instrument from day oneBuild the rejection-reason capture into Stage 3 from the start, not as a later add-on.
Eval before flowDesign the eval for the draft-state failure mode before building the review flow, so that what 'reads as finished but isn't' shapes the visual language instead of following it.
Compare notes
Working on a product that’s still taking shape?
More work