AI is a window, not a wall

Last week I was walking a client through a setup we had just finished. Four steps, one screen, no slides. Somewhere around the third step the CEO cut in:

"Couldn't you have done this without AI?"

The room went quiet in the particular way rooms go quiet when someone asks the question everyone was already thinking. I will be honest — I enjoyed it. It is the best question a buyer has asked me all year. And the answer is uncomfortable.

Most of it, yes. We could have.

Here is why they called us in the first place. Their return rate had climbed four points in two weeks and nobody noticed. Not because nobody was paying attention — because the number lived in four different places, and no single person was looking at all four.

So we built four steps:

1. Pull the data into one place. A cron job and some SQL.
2. Raise a flag when a threshold is crossed. One alarm rule.
3. Check what competitors were charging. A browser script.
4. Read four thousand support tickets written in free text. A model.

Three of those four are not AI, and they should not be. If a language model is deciding whether 4.1 is bigger than 4.0, you have hired a philosopher to read a traffic light. He will give you a thoughtful answer, appropriately hedged, occasionally wrong — and he will bill you for the reasoning.

This is not a small mistake. It is the mistake.

I used to think misplacing AI was a matter of taste. Then the numbers started coming in.

In June 2025, Gartner predicted that more than 40% of agentic AI projects will be canceled by the end of 2027 — killed by escalating costs, unclear business value, or inadequate risk controls. That came out of a poll of more than 3,400 organizations. The line I keep returning to is from Gartner analyst Anushree Verma: "Many use cases positioned as agentic today don't require agentic implementations." Gartner also has a name for the supply side of this — agent washing, the rebranding of existing chatbots and RPA as agents. Their estimate is that of the thousands of vendors selling agentic AI, roughly 130 are real.

A month later, MIT's Project NANDA published The GenAI Divide: State of AI in Business 2025, reporting that despite thirty to forty billion dollars of enterprise spending, roughly 95% of generative AI pilots produced no measurable return. Not bad returns. No measurable return at all.

And McKinsey's State of AI work keeps landing on the same wall: most companies now use AI somewhere, but only about 39% report any EBIT impact at the enterprise level, and the single strongest correlate of impact is not model choice or vendor — it is fundamental workflow redesign.

Read those three together and a pattern shows up that nobody wants to put on a slide: the industry is not failing at AI. It is failing at placement. The models mostly work. They are bolted to the wrong part of the machine.

So why is everyone still doing it?

This is the question I actually get asked, and it is never asked in the meeting. It is asked in the corridor afterwards, quietly: if the failure rate is this well documented, why is my board still asking me for an AI strategy?

The answer is not stupidity. Three forces push at once, and each one is individually rational.

The demo stopped costing what the system costs. You could not demo an ERP without an ERP; an afternoon and a decent prompt now produce something nearly indistinguishable from a working system. The demo used to be evidence. Now it is a costume.

You are buying in a lemon market. Making a claim is free, checking one takes a quarter — and Akerlof won a Nobel for predicting exactly the equilibrium Gartner measured when it counted roughly 130 real vendors among thousands.

The career arithmetic is asymmetric. Nobody has ever been hauled into a board meeting to explain a pilot. Plenty of people have been hauled in to explain the absence of one.

I take that argument apart properly in a separate piece — nobody gets fired for buying AI — because it deserves more room than it gets here.

What matters for the rest of this article is the way out. Akerlof did not end his paper with outrage. He ended it with remedies — warranties, certification, reputation — mechanisms that make quality cheap to verify. That is all a good test is. What follows is a warranty you can issue yourself, in about ninety seconds, without asking a vendor's permission.

The oldest rule in the field

None of this is a new discovery, which is the genuinely embarrassing part.

In 2016 — before any of the current hype existed — Google published Rules of Machine Learning, Martin Zinkevich's field guide of 37 rules. Rule number one, out of thirty-seven, is this:

Don't be afraid to launch a product without machine learning.

Not rule thirty. Rule one. Before anything about features, objectives, or training. The first thing an ML engineer at Google was told was: check whether you need this at all.

Eight years later, Anthropic's Building Effective Agents — written by people whose commercial interest is you using more of their model — says the same thing in the same position: "We recommend finding the simplest solution possible, and only increasing complexity when needed." It goes on to draw the distinction most decks skip entirely: workflows run through predefined code paths, agents let the model direct itself, and agentic systems "trade latency and cost for better task performance."

So the people who build these systems and the people who build the models have been saying this for a decade. The layer in between — the deck, the vendor, the budget line — is the one that forgot.

The wall and the window

Here is how I think about it now, and it is the only mental model I have found that survives contact with a real project.

A building needs walls and it needs windows. Brick is cheap, boring, and utterly dependable: put a load on it and it holds, every time, in the same way. Glass is expensive, beautiful, and does the one thing brick cannot — it lets what is outside come in.

Determinism is your load-bearing wall. AI is your glass.

Nobody argues about this in construction. But in software we have somehow started building load-bearing walls out of glass, because glass photographs better. And then we are surprised when the building settles.

What makes AI genuinely irreplaceable is narrower and stranger than most people think. I would call it the naming problem: you can only write a rule for a category you have already named.

Look at those four thousand tickets. "Small fits." "Runs narrow." "Tighter than usual." Three sentences, one meaning, not a single shared word. You cannot write a regex for a synonym you have not thought of — and you cannot think of the synonym, because if you already knew the reason for the returns, you would not be looking for it.

That is the whole thing. Rules answer the questions you have. Models find the questions you do not. Determinism is for what you can name; AI is for what you cannot. And there is a lot you cannot: somewhere between 80 and 90 percent of what a company produces is unstructured — tickets, calls, emails, documents — and most of it is never read by anything at all.

Three questions that decide whether

This is the test I run, and it takes about ninety seconds per step.

1. Is the input unstructured? Free text, speech, images, a PDF someone scanned crooked. If the input already has columns, you do not have a model problem — you have a query.

2. Can the branches be enumerated in advance? If you can write the full list of possible outcomes on a whiteboard, then write them in code. A finite branch set is a rule engine wearing a costume.

3. Is the output language? A summary, an explanation, a draft, a placement into fuzzy categories. If the output is a number or a boolean, be extremely suspicious of any model standing near it.

Three noes: write code. Two noes: still write code. One no: come argue with me, and bring the data.

Two questions that decide whether it is worth it

The three above tell you whether AI can do the job. They tell you nothing about whether you should ship it. This is the part my LinkedIn version left out, and the part that has cost me the most.

4. What does being wrong cost, and can you take it back? A misclassified ticket is a shrug. A misworded collections email to your largest account is a phone call from your CEO. Cheap and reversible, let it run. Expensive or irreversible, the model still does the work — but a human signs. That is not bureaucracy, that is where approval gates belong. Note the shape of the answer: it is never "don't use AI here." It is "don't use AI unsupervised here."

5. How many times must this judgment be made? Four thousand tickets is worth a model. Twelve tickets a month is worth an afternoon and a coffee — and you will learn more from reading them yourself than the model would ever have told you. My rule of thumb: if one person could clear a month's volume in an afternoon, you do not have an AI problem, you have an afternoon. This is unit economics, and it is the arithmetic your CFO is going to ask for anyway.

The arithmetic nobody puts in the deck

There is a harder reason to keep models out of the wall, and it is multiplication.

Chain steps together and their reliabilities multiply. Not average — multiply. A step that is right 95% of the time is excellent for a language model. Put five of them in a row and the chain works 77% of the time. Put twenty in a row and you are at 36%.

Think of connecting flights. Every leg is 95% on time, which sounds fine. Book five legs and you have roughly a three-in-four chance of making the wedding. Book twenty and you should have driven.

This is why placement is not an aesthetic preference. Every step you hand to a model is a step you move from 1.0 down to 0.95-something, and those numbers compound down the whole chain. A deterministic step is not merely cheaper — it is a 1.0 in the product. It dilutes nothing downstream.

Then there is the boring cost, which is somehow always the one that kills the project in month four: a SQL query answers in milliseconds for a rounding error, while a model call answers in seconds and bills per token, every single time, forever. Costs that nobody put a ceiling on have ended more pilots than bad accuracy ever has.

The opposite mistake, which is just as expensive

I should be fair, because this test cuts both ways and I have watched teams fail on the other side of it.

Somewhere there is a team maintaining forty regexes for the phrase "runs small." Then sixty. Then someone adds Turkish. Then the person who wrote them leaves, and now nobody dares delete any of them, so the file only grows.

A rule list that grows every week is a model with a worse interface, no eval set, and one maintainer who is quietly updating their CV. If your team is hand-maintaining a taxonomy of human phrasing, you have already built a bad model. Go buy a good one.

The test says "use a model" exactly as often as it says "don't." That is what makes it a test rather than a preference.

Read the bill of materials

Nobody calls their car "a GPS car." The GPS is one component of maybe four hundred, and it happens to be the one you look at, so it does not get to name the vehicle.

Our returns project was four steps, one of which used a model. That is 25% AI and 100% called AI. The market now has names for that gap. Gartner calls it agent washing. The SEC calls it AI washing, and in March 2024 it fined two investment advisers — 225,000 and 175,000 dollars — in the first enforcement actions of their kind, for claiming AI capabilities they did not have.

I am not suggesting your internal deck is securities fraud. I am saying a regulator has now put a price on the distance between what a system does and what it is called, and that price only goes one direction.

So here is the discipline, and it is not complicated. List every step. Label each one: movement, rule, integration, or judgment on unstructured input. Then defend every step you labeled judgment, one at a time, out loud.

If I cannot say in one sentence why a step needs a model, the step does not need a model. That sentence has killed more of my own designs than any client ever has.

Where we sit, and why we shipped it as four things

Fair question to put to someone who sells an AI platform: where do you land on your own test?

The honest answer is that the shape of our company is the shape of this argument. If most of the work inside an automation is not AI, then most of the products cannot be either.

• Piperails moves data — twenty-one-odd sources into six destinations, on a schedule. There is no language model inside it, because pulling yesterday's orders out of an ERP is transport, and transport is a solved problem with a cron expression.
• DataRails answers the questions you can already name. SQL, a semantic layer, dashboards. No model. If you know the question, you want a query, not a conversation.
• RoboRails drives a real browser on a schedule and asserts pass or fail — competitor prices, the checkout funnel, the login that broke at 3am. No model there either. A monitoring step is deterministic or it is worthless as a monitor.
• PromptRails is where the models go. And even there, most of what we build is not the model: it is the apparatus around it — guardrails, PII masking with policy and audit, human approval on the steps that deserve it, and a trace of every decision the thing made.

Three of those four contain no language model at all. That is not modesty. That is this test applied to ourselves, and it is the reason they shipped as separate products instead of one box with AI written on the side.

One detail is the whole argument in miniature. Piperails exposes an MCP server, so a model can call the pipeline. That direction is deliberate. The model asks the pipeline for the number; the pipeline never asks the model whether 4.1 is bigger than 4.0.

And the one model we published on its own — piimask, 0.5B parameters, running on your own hardware — does exactly one job: finding personal data in free text, in English and Turkish. Unstructured input, branches nobody can enumerate, language output. It passes all three questions, which is precisely why it exists as a model instead of a regex file that someone would have to keep feeding forever.

How to actually answer the CEO

Here is the thing I believe now, and it took a few uncomfortable meetings to learn: "most of this isn't AI" is not a confession. It is the strongest thing you can say in that room.

Because the CEO was not really asking "is AI necessary?" He was asking "am I being sold something?" The moment you draw the line yourself — these three steps are plumbing, this one step needs a model, and here is precisely why nothing else does — you stop being a vendor and start being the person in the room doing the arithmetic.

The version I would hand you to use verbatim:

"Three of these four steps are ordinary engineering, and we built them that way deliberately. One step reads four thousand sentences that no rule can catch — that is the one that needed a model. If we had used a model for the other three, you would be paying more for a system that is less reliable."

Nobody has ever argued with that. The projects that die in the room are the ones where the honest answer is "well, AI is more flexible."

Monday morning

Take your AI portfolio — every project, every pilot, every vendor demo currently on someone's desk — and for each one write three things:

1. The steps. Not the goal. The steps. It is almost always between four and seven.
2. A label for each step: movement, rule, integration, or judgment on unstructured input.
3. One sentence of justification for every step you labeled judgment.

Two things will happen, and both are good news.

Some projects will collapse into a cron job and a query. Ship those this quarter — they are the fastest ROI in the building, and they have been sitting there disguised as something expensive.

And one or two will survive with a genuinely irreplaceable model step in the middle. Those are your real AI projects. Fund them properly, gate them, trace every decision they make — and stop spreading the budget across the ones that were never going to need a model in the first place.

Glass is the most expensive material in the building and the only one that does what it does. That is exactly why you do not build the wall out of it.

So — on your list, how many are AI because the work demands it, and how many because the slide did?

References

• George A. Akerlof, "The Market for 'Lemons': Quality Uncertainty and the Market Mechanism," Quarterly Journal of Economics, 1970.
• Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027," press release, June 25, 2025.
• MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, July 2025.
• McKinsey & Company, The State of AI survey series, QuantumBlack.
• Martin Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google, 2016.
• Anthropic, Building Effective Agents, engineering blog, December 2024.
• U.S. Securities and Exchange Commission, "SEC Charges Two Investment Advisers with Making False and Misleading Statements About Their Use of Artificial Intelligence," March 18, 2024.
• IDC, Data Age 2025 — on the share of enterprise data that is unstructured and unanalyzed.

---

Want to run this test on one of your workflows? Book a 30-minute call and we will label the steps together — including the ones that turn out not to need us.