The Day After the Demo

Dana: Picture the meeting. A small team shows an AI assistant answering questions about insurance claims. It finds the right procedure, summarizes the file, drafts a note to the customer. People are impressed, and somebody asks the obvious question: how soon can we roll this out?

Marcus: And six months later it’s still a pilot. Nobody decided it failed. It just never became something people could rely on.

Dana: That meeting is where today’s book begins. It’s called The Day After the Demo: Getting AI Out of the Pilot and Into the Work, by Richard White of RJL Solutions. Richard has led data and AI at McKinsey, New York Life, the Federal Reserve Bank of New York and Citigroup. I’m Dana.

Marcus: And I’m Marcus. A quick note before we start: we’re AI narrators. This script was written from the book, and every example we use is a composite from the book, not a client story. The book itself is free to download at rjlsol.com/book.

Dana: So let’s start with the line that sets up the whole thing. Marcus, what does a demo actually prove?

Marcus: Less than people think. The book puts it this way: the demo proves a narrower claim than the one you are about to fund. A demo shows a model producing useful output on chosen cases, under conditions somebody set up. What the room hears is that the business can rely on that output, at scale, with ordinary users and changing data.

Dana: And the gap is in the conditions.

Marcus: Right. Somebody picked the documents. The presenter knew what every term meant. A service account could reach everything. A plausible answer ended the exercise. And when something broke, the builders fixed it on the spot. Daily operation supplies none of that.

Dana: There’s an exercise in the book I really like. Follow one request all the way through.

Marcus: Take a claims examiner who asks for the status of a claim and the procedure that applies. For that answer to be right, five things have to be true. The information is right, meaning the person linked to the claim is the right person in the right role. The authority is right: she may read the case, but that doesn’t mean an agent acting for her may pay it or contact the customer. Someone owns the result, so when a definition is disputed, a named person can decide. The value is real, which means useful work actually went up once you count the checking and the corrections. And it can be run. Someone takes the incident at two in the morning.

Dana: And Richard’s test is simple. Wherever the explanation turns into an assumption, the release still has work to do.

Marcus: He says he asks teams to do this before they ask for more money. It works as well in a bank as in a portfolio company.

Dana: So why does this keep happening? The book gives three reasons.

Marcus: First, the funding model rewards the demo. Innovation budgets pay for proofs of concept, operating budgets pay for running things, and the new capability falls in between. Second, the work is split across functions that rarely design together: the AI team, the data team, security and the business each do their part well, and the whole thing still has no owner. And third, AI gets bolted onto the existing job. The examiner still does every old step, and now she checks the machine too.

Dana: Which brings us to Part One, which is about owning the outcome. The first chapter says start from a decision and a baseline, not from a list of ideas.

Marcus: The familiar pattern is a steering group that collects ideas, plots them on a two by two, and funds the top ten as proofs of concept. Six months later there are ten demos and none in daily use. The book’s alternative is to pick work people repeat every day, with an owner who can change it, and measure the starting point before you build anything.

Dana: And then do the economics honestly. There’s a worked example.

Marcus: A hundred and twenty thousand eligible cases, sixty percent adoption, six minutes saved per case after review, sixty five dollars an hour. That releases seven thousand two hundred hours, worth four hundred sixty eight thousand dollars at labor cost. Take off two hundred twenty thousand dollars of running cost and you get about two hundred forty eight thousand dollars of net capacity value.

Dana: But the book is careful about what that number means.

Marcus: Very careful. Time saved is capacity, not cash. It only becomes a saving when staffing, overtime or contractor spend actually changes. The line I remember is: keep that distinction visible, or finance will draw it for you later, and less kindly.

Dana: Chapter Two is about owners with real authority, and it has one of my favorite scenes. A claims assistant waits six weeks on one word.

Marcus: The word is open. Claims operations says a claim is open while it still needs an examiner’s work. Finance says it stays open while a payment is unsettled. Every Thursday the working group asks for more analysis, engineering produces another reconciliation showing three hundred and twelve claims that are open under one definition and closed under the other, and nothing moves.

Dana: And the fix isn’t another count.

Marcus: No. It’s deciding which purpose the first release serves and who can approve the definition for that purpose. Claims can own the operational meaning, finance keeps its financial meaning, and engineering encodes both with a reconciliation where they meet. Nobody has to pretend the other side is wrong.

Dana: Chapter Three makes a point I don’t hear often enough. Human review is a job, and it needs capacity.

Marcus: The book does the arithmetic. If a thousand items arrive a day and each takes three minutes to review, that’s fifty hours of review every day. If each reviewer has five hours for it, you need ten reviewers’ worth of time. And it doesn’t become eight because the launch plan lists eight names.

Dana: I laughed at that one.

Marcus: It’s funny because it’s true. And the book goes further. If the only button on the screen says Accept, you’ve built confirmation, not judgment. Reviewers need to be able to reject, correct, ask for more, and stop an unsafe path.

Dana: Let’s jump to Part Three, because a lot of listeners will have this question. Should we use ChatGPT or Claude? Should we be on Bedrock, Azure or Google?

Marcus: The book’s answer is that those are really four questions hiding in one. Which assistant employees use every day. Which models do the work. How you reach those models, whether that’s the vendor directly or through your cloud. And where your agents and automations actually run.

Dana: And each one has a different owner and a different shelf life.

Marcus: Exactly. For the assistant, start with the one native to your productivity suite, because it already sees your documents and permissions. For models, there’s no single best one and the leader changes within months, so test them on your own cases, approve at least two providers, and keep that decision easy to reverse. For access, your strategic cloud’s catalog is usually the right default, and one model is rarely a good reason to take on a second cloud. And for the runtime, which lasts for years, that’s where you apply the hard gates and a production-shaped proof.

Dana: Part Four is data the business can trust, and the headline is that a data product is a maintained promise.

Marcus: There’s a great scene. At ten forty in the morning an examiner asks for a claim’s current status. The assistant reads the published status table and says the claim is open and approved. The table is accurate as of last night. At nine fifteen that morning the claim was put on hold for a fraud review.

Dana: And nothing was broken.

Marcus: Nothing. No corrupt field, no failed pipeline. The table was built for a monthly report and does that job well. The mistake was serving a daily snapshot as if it were the state of the claim right now. Calling the table gold or certified wouldn’t have changed the answer. So the book says start the contract with the consumer and the decision: what may they rely on, how current must it be, and what happens when that condition isn’t met.

Dana: The new edition also adds a worked invoice example.

Marcus: It does. One invoice, a thousand dollars of charges, a hundred of credits and four hundred of payments, so five hundred outstanding at the first cutoff. Then a payment of two hundred arrives and the current balance becomes three hundred. The earlier record stays as a dated version. A late payment changes what you know now without quietly rewriting what an earlier report knew at its cutoff. And a dated balance can never authorize a change to a live invoice.

Dana: Which leads nicely into Part Five, agents you can answer for. Chapter Thirteen has the scene that made me wince.

Marcus: The duplicate note. The assistant has earned permission to write an internal note once an examiner approves it. She approves. The claims system accepts the note, but the connection drops before the assistant gets the confirmation. The screen says failed. She clicks again. Now the claim has two notes.

Dana: And next time it might be a payment.

Marcus: That’s the point. The book says a timeout tells you confirmation didn’t arrive. It doesn’t tell you whether the change happened. So you give each intended action one identity, you tie the approval to its exact content, you check the state at the moment of execution, and when you don’t know, the honest message is not yet confirmed, not failed. Failed invites the second click.

Dana: There’s also a line about autonomy that I think matters a lot right now.

Marcus: Autonomy is not the goal. The outcome is, at a level of risk the owner accepts. People set the boundaries at design time, watch for drift while it runs, and step in where the consequence of an error calls for it. And the book connects this to business models. Take a mutual life insurer whose advice reaches customers through its financial professionals. An AI that advised customers on its own wouldn’t be a better workflow. It would be a different company.

Dana: Chapter Fourteen is about permissions, and it opens with two quiet failures.

Marcus: The first is a scanned letter in a claim file with a paragraph addressed to the AI assistant, saying the examiner is authorized to see the claimant’s other claims. The second is an examiner who moves teams on a Friday, and on Monday the assistant helpfully resumes Thursday’s conversation, showing a summary from a file she can no longer open.

Dana: And neither one is fixed by better instructions to the model.

Marcus: Right. The prompt is not an authorization boundary. The systems that hold the data have to enforce permission for the person asking, at the moment they ask. A document must never grant a privilege. And every stored copy, the index, the cache, conversation memory, has to respect today’s permissions, not yesterday’s.

Dana: Let’s finish with Part Six, delivery and adoption. Chapter Eighteen has the trial that went beautifully.

Marcus: Twelve experienced volunteers, managers clearing their time, the engineer on hand. Six weeks later, glowing results and a rollout. Three months after that, use is flat, because the wider rollout reached newer staff, harder cases and a normal workload. The trial result was real for the twelve. Nobody tested whether it held for the other three hundred.

Dana: And the release chapter has a warning about averages.

Marcus: My favorite. The model provider announces a version is going away. The team reruns its tests and the average quality score goes up, from ninety one to ninety three percent. Looks routine. But hidden under that average, the new version twice answers whether a claim is covered, which it’s not allowed to do, and it reacts differently to a timeout. So the book’s rule is that no average makes up for a critical failure. Test those one at a time.

Dana: And then prove the handover.

Marcus: Transfer is proven, not declared. The receiving team restricts the capability, rolls back a release and revokes a user, using only the runbooks, while the builders watch and don’t help. Anything that needs a builder to step in is a gap.

Dana: If you had to give listeners one thing to do this week, what would it be?

Marcus: The book ends with it. Take one request to the people who own its result, and follow it until every assumption that matters has an owner and a test. Then fund one complete workflow, with real users, approved information, enforced authority and a measured outcome, and let its results decide whether you expand it, reshape it, or stop.

Dana: The book is The Day After the Demo, by Richard White, and it’s free at rjlsol.com/book. Thanks for listening.

Marcus: Thanks, everyone.