The case study nobody publishes: when the tool got switched off

Every legal-tech case study you’ve ever read is a success story. The tool was deployed, adoption soared, time was saved, everyone’s delighted, here’s a quote from a happy partner. I understand why - nobody markets their failures. But the genre is dishonest by omission, because the most useful lessons in this field live in the case studies nobody publishes: the ones where the tool got switched off.

So here’s one. Details changed enough to protect the people involved, but the shape is real and the lessons are exact.

The setup

A good tool. Genuinely good - it did a real, painful, high-volume task well, and in testing it saved meaningful time. The technology worked. The pilot numbers were strong. The fee-earners who tried it in the controlled pilot liked it. Everything you’d want from a green light was there, and the firm rolled it out properly: training, a launch, a champion, all the right moves.

Six months later it was switched off. Not dramatically - no disaster, no incident. Usage just decayed to near-zero, the renewal couldn’t be justified, and quietly the thing was retired. A good tool, deployed competently, dead within two quarters. Those are the most instructive failures precisely because nothing obvious went wrong.

What actually killed it

When I dug into why - and you have to dig, because the official story is always vague - it wasn’t one thing. It was three, and each one is a lesson.

The time it saved went to the wrong person. The tool saved time at the firm level, but on any given matter, using it cost the individual fee-earner a few extra minutes of setup and checking versus just doing the task the way they always had. Aggregate benefit, individual cost. From the firm’s spreadsheet the tool was a win. From the desk of the person being asked to use it, on a specific busy afternoon, it was a small chore with the payoff going to someone else. The user decides adoption, not the spreadsheet, and the user quietly opted out.

The checking ate the saving. The tool’s output was good but not checkable enough. To use it responsibly, fee-earners had to verify the output carefully - and because the tool didn’t show its work well, “carefully” meant “almost entirely.” Once you’re checking almost everything, the tool hasn’t saved you the work, it’s added a step. The savings were real in testing, where people trusted the pilot conditions, and evaporated in production, where they were professionally on the hook and checked accordingly.

There was no owner of the boring middle. The launch had a champion, but the champion’s job was the launch. After the launch - when the real adoption work happens, the answering-the-same-question-forty-times, the fixing-the-rough-edges, the chasing-the-team-that-stopped-using-it - nobody owned it. The enthusiasm faded on schedule, and there was no one whose actual measured job was to carry the tool across the gap from “deployed” to “habit.” So it never crossed it.

The lesson the success stories hide

None of those three failures was about the technology. The tool was good. It died of incentive design, of insufficient checkability, and of an unowned middle - exactly the things a success-story case study never mentions, because the successful version solved them quietly and nobody thought to write down the solving.

That’s the deep problem with only publishing wins. The wins make it look like the technology was the variable - “we bought a good tool and it worked.” But the good tool fails just as easily as the bad one if the time saving lands on the wrong person, if the output isn’t checkable, or if nobody owns the boring middle. The variable was never mainly the technology. The success stories systematically hide that, and so a whole industry keeps re-learning it the expensive way.

What I do differently because of it

That switched-off tool changed how I work more than any success ever has. Now, before building or deploying anything, I ask the three questions it failed:

Does the time saving land on the same person being asked to change their behaviour - or are we asking an individual to absorb a cost so the firm can have a benefit? If it’s the latter, the tool will quietly die regardless of its aggregate value, and we need to fix the incentive or expect the death.

Can the user check the output faster than they could do the work themselves? If the responsible way to use the tool is to check everything, there is no time saving in production, whatever the pilot showed. Checkability isn’t a nice-to-have - it’s the difference between a real saving and a theatrical one.

Who owns this in month six, and is that ownership their measured job rather than a favour? The launch is the easy part. If no one’s success is tied to the tool still being used two quarters later, it won’t be.

Why I’m telling you about a failure

Because the success stories have taught the industry the wrong lesson - that good technology is the thing that determines whether legal AI works. It isn’t, and the switched-off tools are the proof. They’re just the case studies nobody publishes, which means the most important lessons in legal AI are the ones the field has agreed not to talk about.

The tool was good. It still died. Sit with that, because it’s the most useful sentence in legal tech, and you’ll never read it in a vendor’s case study.