Hype, Evidence and the Lemon Market

Why this module exists

Executives are told the problem is hype and the solution is scepticism. That advice is useless, because it supplies no method. It only changes the mood in the room — and a sceptical executive with no method makes exactly the same decisions as a credulous one, just more slowly.

This module treats the market as an economic system with knowable properties. Three forces explain almost all of the behaviour you are seeing, and none of them is stupidity. Once you can name them, you can evaluate a claim in minutes instead of quarters, and you can explain your reasoning to a board without sounding like a pessimist.

2.1 What the numbers actually say

Four findings frame the field. Take them together, and take their limits seriously — a course that quotes failure statistics uncritically is committing the error it is warning about.

Gartner, June 2025. More than 40% of agentic AI projects will be cancelled by the end of 2027, driven by escalating costs, unclear business value and inadequate risk controls. Gartner also names the supply-side behaviour — agent washing, the relabelling of existing chatbots and robotic process automation as agents — and estimates that of the thousands of vendors selling agentic AI, roughly 130 are real. Limitation: this is a forecast, not a measurement, and it is drawn substantially from a poll of self-selected webinar attendees.

MIT Project NANDA, July 2025. Roughly 95% of enterprise generative AI pilots produced no measurable business return, against thirty to forty billion dollars of spending. Limitation: "no measurable return" partly measures how few organisations set up measurement in the first place. That is a real finding, but it is a finding about measurement discipline as much as about the technology.

McKinsey, State of AI. The large majority of firms now use AI somewhere, yet only around 39% report EBIT impact at enterprise level, and the strongest correlate of impact is not model choice or vendor — it is fundamental workflow redesign. Limitation: self-reported attribution of profit to a technology is among the least reliable numbers in management research. Treat the ranking of factors as more robust than the level.

FactSet, Q1 2026. Of the 498 S&P 500 companies holding earnings calls, 337 mentioned "AI" — 68%, the highest in a decade against a ten-year average of 103. And companies that mentioned it outperformed those that did not over the following period. Limitation: correlation with share price is not causation, and the composition of the mentioning group is skewed towards technology.

Read the four together and one reading survives all the caveats:

The industry is not failing at AI. It is failing at placement, measurement and follow-through — while the market pays for the word.

That last clause is not cynicism, it is the FactSet data. Once saying the word is rewarded, everything downstream is people responding correctly to a price signal.

2.2 Force one: the demo stopped costing what the system costs

For every enterprise technology before this one, building a convincing demonstration was roughly as expensive as building the thing. You could not demo an ERP without an ERP. You could not fake a warehouse management system in a conference room; the pallets either moved or they did not. The demo was therefore evidence, because producing it required having solved the problem.

A language model severs that link. An afternoon and a well-crafted prompt produce something nearly indistinguishable from a working system, because in both cases the output is a paragraph of confident text.

The demo used to be evidence. Now it is a costume.

Notice who this disarms, because it is counterintuitive. A CFO can read a profit-and-loss statement. An operations director can walk a floor and tell you within ten minutes whether something is real. A CTO can read an architecture diagram and find the lie in it. But nobody in that room, at any level of seniority, can look at a chat window producing fluent paragraphs and tell you whether it will hold up on Tuesday against four thousand real messages.

Seniority normally buys pattern recognition. Here it buys almost nothing — and the pretence that it does is how expensive decisions get made quickly.

The practical response is not to refuse demos. It is to insist that a demo is the bottom of the evidence ladder, and to state out loud what would move it up.

2.3 Force two: you are buying in a lemon market

In 1970, George Akerlof asked a question about used cars that turned out to be about a great deal more: what happens to a market when the seller knows the quality and the buyer does not?

His answer runs as a mechanism, and it is worth following step by step because your own instincts are in it:

Buyers cannot distinguish good from bad, so they will only pay a price reflecting average quality.
Sellers of genuinely good goods, worth more than the average, decline to sell at that price and exit.
Average quality therefore falls, so the price buyers will pay falls further.
More good sellers exit. The market unravels toward the lemons.

Akerlof shared the Nobel Prize for this in 2001. Now place Gartner's estimate beside it — roughly 130 real vendors among thousands claiming agentic capability. That is not a scandal, and it is not a moral failing peculiar to this industry. It is the predicted equilibrium of a market in which making a claim is free and verifying one takes a quarter.

You are the buyer in that market. And here is the uncomfortable part: your instinctive defences — pay the average, hedge, run a small pilot with a small budget — are the textbook-correct response to information asymmetry, and they are precisely the behaviour that produces underfunded pilots that never leave the sandbox. Rational individual caution aggregates into the 95% statistic.

Akerlof did not end his paper with outrage. He ended it with remedies, and there were three:

Akerlof's remedy	Your procurement equivalent
Guarantees	A pre-registered success metric on your data, with a decision rule agreed before the pilot
Certification	Independent standards and audits — ISO/IEC 42001, SOC 2, and the AI Act conformity work of Module 6
Reputation	Reference customers you select, including one that churned

You cannot fix an information-asymmetric market by complaining about vendors. You fix it by becoming cheap to verify — a theme Module 7 turns into a scorecard.

2.4 Force three: the career arithmetic is asymmetric

Keynes wrote the line in 1936, about investment committees, and it has not aged:

Worldly wisdom teaches that it is better for reputation to fail conventionally than to succeed unconventionally.

Count the payoffs honestly inside your own organisation. An AI project that fails is a project that failed; everyone's failed, the market is hard, next item. Not having one, in 2026, is treated as a personality trait. Nobody has been called into a board meeting to explain a pilot. Plenty of people have been called in to explain the absence of one.

This is not a feeling. Two surveys make it measurable:

Dataiku's survey of chief executives found that 74% believe they could lose their job within two years if they do not deliver measurable AI gains.
WRITER, with Workplace Intelligence, surveyed 2,400 employees and C-suite leaders in April 2026: 75% of executives said their own company's AI strategy is "more for show" rather than genuine internal guidance, naming public relations and investor relations as the reason it exists. Thirty-nine percent had no formal plan for turning any of it into revenue.

Three quarters, admitted, by the people who commissioned the strategy.

Stack the three forces and the behaviour stops looking like madness and starts looking like an equilibrium: claims that are free to make and expensive to verify, evaluated by people who structurally cannot verify them, inside a system where being conventionally wrong costs nothing.

The leadership consequence is the useful part, and it is small and local. You cannot change the market. You can change what is rewarded in the one room you control — which is why Module 9 treats incentives as a governance instrument rather than a soft topic.

2.5 The evidence ladder

Here is the method the module promised. Every claim — from a vendor, a consultant, or your own team — sits on one of five rungs. The rung determines the size of commitment it justifies.

Rung	What it is	What it justifies
1. Demonstration	A scripted walkthrough on the vendor's data	A second meeting. Nothing else
2. Case study	A written account you cannot independently check	Shortlisting
3. Reference	A customer you selected and called yourself	A paid pilot
4. Pilot	Your data, your volumes, a metric and decision rule fixed in advance	A production decision
5. Production evidence	A deployed system with published error rates over time	Scaling, and reuse of the pattern elsewhere

Two rules make this operational.

What each rung of evidence actually buys. Most vendor conversations start at the bottom and are treated as though they started near the top.

Rule one: name the rung out loud. Not as an accusation — as bookkeeping. "This is a rung-one claim, which is fine at this stage; here is what would take it to three." It converts a sales conversation into a shared workplan, and vendors who cannot climb the ladder disqualify themselves without an argument.

Rule two: pre-registration is what separates rung four from rung one. A pilot whose success metric is chosen after the results are in is not a pilot. It is a demonstration with a longer runway, and it will be interpreted favourably by whoever sponsored it. Write down the metric, the baseline, the sample and the decision rule before anything runs, and have someone who is not the sponsor hold the document.

2.6 Reading a claim in the meeting

Four questions, in order. They take about two minutes and work without any technical background.

"What is the denominator?" — 40% better than what, measured over how many cases? A percentage without a base is decoration.
"Whose data was it on?" — Yours, theirs, or a public benchmark? Benchmark performance transfers to your documents far less reliably than anyone hopes.
"What happened to the cases it got wrong?" — A vendor who cannot describe their failure modes has not looked. This question separates rungs one and two from three and above faster than any other.
"Who else has stopped using it, and why?" — Ask for a reference that churned. The refusal is itself information.

Vantis Diagnostics — a proposal on the ladder

A vendor pitches Vantis an "AI agent" to reduce claim denials in the US lab, opening with a live walkthrough: a denial is pasted in, an appeal letter appears, the room is impressed.

The head of revenue cycle applies the ladder rather than her instincts.

Where it sits: rung one. The denial was one the vendor chose, on the vendor's environment.

What she asks:

Denominator: "You said 30% fewer denials. Across how many claims, over what period, and at which payers?" The answer — one client, one payer, three months — does not invalidate the claim, but it prices it.
Whose data: she requests a rung-four pilot on 500 of Vantis's own denials, including the two payers that generate most of the volume.
Failures: "Describe the appeals it drafted that were rejected." The vendor's answer is specific and unflattering, which raises her confidence rather than lowering it.
Churn: the vendor supplies a customer who stopped. That call reveals the real finding — the tool worked, but the client could not staff the review step, so letters went out unread until they paused it.

The decision: a paid pilot, pre-registered — 500 denials, measured on overturn rate against the trailing twelve-month baseline, decision rule agreed in writing, and a named biller with capacity to review. The churned reference changed the design of the pilot, not the decision to run it. That is what rung three is for.

Exercise — Put a live proposal on the ladder

Time: 45 minutes. Produces the artefact for this module.

Take the most recent AI proposal that reached your desk — vendor or internal, it does not matter.

List its load-bearing claims. The three or four that, if false, would change your decision. Ignore the rest.
Place each on the ladder and mark the rung.
Find the weakest load-bearing claim — the one carrying the most weight from the lowest rung. This is your project's actual risk, and it is rarely the one on the risk register.
Write the single question that would move it up one rung, and send it today.
Draft the pre-registration for the pilot you would run: metric, baseline, sample, decision rule, and who holds the document.

Keep the ladder. It applies unchanged to the vendor scorecard in Module 7 and to the evidence dimension of the capstone rubric.

Self-check

A vendor reports "a 40% improvement." Name the two questions that most efficiently establish what that means.
Why does individually rational buyer caution aggregate into the 95% pilot-failure statistic?
What single procedural step distinguishes a pilot from a demonstration, and why must it come first?
Your team says a competitor "has already deployed this." What would you need to see to place that claim above rung two?
Which of Akerlof's three remedies is most available to you as a buyer this quarter, and what would you do with it?