Where AI review breaks down: the long tail of edge cases

Ask a legal AI tool to review a hundred standard commercial leases and it will, these days, do a remarkably good job. It will find the break clauses, pull out the rent review dates, summarise the repairing obligations and do it in a fraction of the time a trainee would take. Then document seventy-three turns out to have been varied by a side letter nobody uploaded, and document ninety-one has a schedule that refers to a plan that isn’t attached, and the tool reports both with exactly the same calm confidence as the other ninety-eight. That is where AI review breaks down. Not on the average document, which it has largely solved, but on the long tail of edge cases - which is, inconveniently, where most of the legal risk lives.

The average document doesn’t exist

Every review exercise is pitched as if the documents were a uniform population. They never are. A real data room is a geological record of how a portfolio was assembled: leases drafted by different firms in different decades, variations layered on top, licences that quietly change the permitted use, scans of scans with handwritten amendments in the margin. The “typical” lease is a statistical fiction. What you actually have is a cluster of ordinary documents surrounded by a fringe of strange ones, and the fringe is always bigger than anyone expects.

I spent years building review tools for real-estate lawyers, and the lesson that took longest to sink in was this: performance on the ordinary documents tells you almost nothing about how a tool will behave on the strange ones. They are different problems. A system can be excellent at the first and quietly hopeless at the second, and a headline accuracy figure blends the two together until neither is visible.

What the tail actually looks like

It helps to be specific, because “edge cases” sounds like a rounding error and it isn’t. The tail is the lease whose operative terms have been overridden by a deed of variation sitting in another folder. It’s the clause that says “save as provided in the Third Schedule” when the Third Schedule is missing. It’s the definition that lives in a superior lease the tool has never seen. It’s manuscript amendments, struck-through text that a scanner has rendered as ordinary text, and bespoke drafting by a lawyer who had strong personal views about how a rent review ought to work.

None of these are exotic. On any portfolio of meaningful size, they’re guaranteed. And each of them shares a nasty property: the document looks complete. Nothing on the page tells a model that the answer it has confidently extracted has been superseded somewhere else.

Why the tail is where the risk lives

Here’s the part the industry tends to skip. Clients don’t pay lawyers to review documents because of the ordinary clauses. Ordinary clauses rarely cause problems, precisely because they’re ordinary - everyone knows what they mean and the market has priced them in. Claims, disputes and nasty surprises come from the unusual: the break clause with an odd precondition, the user restriction that kills the deal, the variation that changes everything.

So there’s an asymmetry. A tool that handles nineteen documents out of twenty perfectly and misreads the strangest one hasn’t done 95% of the job. It has done the easy part of the job and left the important part looking as though it had been done too. That’s arguably worse than doing nothing, because a lawyer reviewing everything by hand at least knows they haven’t been given the answer yet.

Absence is the hardest thing to review

The deepest version of the problem is that a great deal of legal review is about what isn’t there. A good real-estate lawyer reads a lease and notices there’s no landlord’s covenant to repair the structure, or that the rent review clause has no mechanism for resolving a disagreement, or that the alienation provisions are oddly silent on group sharing. Spotting the missing thing is a core professional skill, and it’s one that extraction-based tools are structurally bad at. You can’t extract a clause that doesn’t exist.

This is why I’m sceptical of review products that present their output as a neatly filled-in table. A table rewards finding things. It has nowhere to put “this should be here and isn’t”, and so the most important observation a lawyer can make about a document is the one the interface silently drops.

Designing for the tail

The answer isn’t to wait for models that never miss. It’s to design the product around the fact that they will. In practice that means a few unglamorous things.

First, treat “this document is unusual” as a primary output, not a footnote. A tool that confidently handles ninety documents and puts the other ten in front of a lawyer with a clear reason is far more valuable than one that has a go at all hundred. Routing is a feature, not an admission of failure.

Second, build document-level sanity checks. Does the schedule referred to actually exist? Do the pages run in sequence? Does this lease mention a deed of variation we don’t have? These checks are boring and cheap, and they catch a remarkable share of the tail before it causes harm.

Third, test on the tail deliberately. When you evaluate a tool - or build one - don’t measure it on a random sample. Assemble the ugliest documents you can find and measure it on those. That’s the test that predicts how it will behave on a real matter.

The temptation in legal AI is to judge a review tool by its average performance, because averages are easy to measure and flattering to report. But nobody remembers the average. A lawyer remembers the one document the tool got confidently wrong, and the trust that went with it. The products that win this market won’t be the ones that are best on the ordinary documents - most will be good enough there soon. They’ll be the ones that know when a document isn’t ordinary, and say so.