Placement: The Five-Question Test

Why this module exists

Most organisations do not have an AI problem. They have a placement problem.

The failure mode is rarely that a model could not do the job. It is that a model was assigned a job that a database query, a scheduled script, or a single rule could have done more cheaply, more reliably, and without a governance review. The result is a system that costs more, breaks more often, and cannot be explained to an auditor — doing work that was never hard.

This module gives you a repeatable test for that decision. It takes about ninety seconds per step, requires no technical background, and produces two artefacts you can put in front of a board.

3.1 The wall and the window

A building needs walls and it needs windows.

Brick is cheap, boring and utterly dependable. Put a load on it and it holds, every time, in the same way. Glass is expensive, beautiful, and does the one thing brick cannot: it lets what is outside come in.

Determinism is your load-bearing wall. A model is your glass.

Nobody argues about this in construction. In software we have started building load-bearing walls out of glass, because glass photographs better — and then we are surprised when the building settles.

The useful question is therefore not "can AI do this?" Modern models can do a great deal, including much they should not be trusted with. The useful question is "is this a wall or a window?"

The naming problem

What makes a model genuinely irreplaceable is narrower and stranger than most executives assume. State it as a rule:

You can only write a rule for a category you have already named.

Consider a real example. An online retailer's return rate rises four points in two weeks. The reason sits in four thousand free-text support messages, expressed like this:

• "Small fits."
• "Runs narrow."
• "Tighter than usual."
• "Had to send back for the next size."

Three of those four sentences share no words at all. You cannot write a pattern-match for a synonym you have not thought of — and you will not think of it, because if you already knew why returns had risen you would not be searching.

This is the boundary:

Rules answer the questions you have. Models find the questions you do not.

Determinism handles what you can name. A model earns its place where naming is the problem itself.

3.2 Questions 1–3: does a model fit?

The first three questions establish whether the shape of the work matches what a language model is for. They are about the problem, not the budget.

Question 1 — Is the input unstructured?

Free text, speech, images, a PDF someone scanned at an angle, a handwritten delivery note. If the input already has columns and types, you do not have a model problem — you have a query.

The trap: data that looks unstructured because it lives in a bad system. A CSV export with inconsistent column names is structured data with a cleaning problem. Fix the pipe, do not hire a model to read the mess.

Question 2 — Can the branches be enumerated in advance?

If you can write the complete list of possible outcomes on a whiteboard, write them in code instead. A finite branch set is a rule engine wearing a costume.

"Approve, reject, or escalate" is three branches. That is a rule. "Summarise why this customer is unhappy" has no enumerable branch set at all. That is a model.

Question 3 — Is the output language?

A summary, an explanation, a draft, a classification into fuzzy categories, a translation. If the output is a number, a date, or a true/false value, be extremely suspicious of any model standing near it.

Scoring. Three "no" answers: write code. Two: still write code. One "no": the case is arguable, and you should be able to state the argument in a sentence.

3.3 Questions 4–5: is it worth doing?

The first three questions tell you a model can do the work. They say nothing about whether you should deploy it. Two further questions decide that, and they are the ones most often skipped.

Question 4 — What does being wrong cost, and can you take it back?

Sort every candidate step on two axes: the cost of an error, and its reversibility.

A misclassified support ticket is a shrug. A mistoned collections email to your largest account is a phone call from your chief executive. Note the shape of the answer: it is never "do not use AI here." It is "do not use AI unsupervised here." The model still does the work; a named human signs.

This is what an approval gate is for, and Module 5 designs them properly.

Question 5 — How many times must this judgement be made?

Four thousand tickets justify a model. Twelve tickets a month justify an afternoon and a coffee — and you will learn more reading them yourself than any summary would tell you.

Rule of thumb. If one person could clear a month's volume in an afternoon, you do not have an AI problem. You have an afternoon.

This question kills more proposals than the other four combined, and it kills them honestly. It is also the question your finance director will ask first, so it is better to arrive with the answer.

3.4 The arithmetic of chained steps

There is a harder reason to keep models out of load-bearing positions, and it is multiplication.

When steps are chained, their reliabilities multiply. They do not average.

A step that is correct 95% of the time is excellent performance for a language model on a judgement task. Chain them:

Think of connecting flights. Every leg is 95% on time, which sounds fine. Book five legs and you have roughly a three-in-four chance of making the wedding. Book twenty and you should have driven.

This is why placement is not an aesthetic preference. Every step handed to a model moves that step from 1.0 to 0.95-something, and those numbers compound along the whole chain. A deterministic step is not merely cheaper — it is a 1.0 in the product. It dilutes nothing downstream.

Two design consequences follow, and you should be able to state both:

1. Shorten model chains. Prefer one model step doing a well-specified job over five model steps reasoning their way along. Length is risk.
2. Put deterministic checks between model steps. A rule that validates the model's output before the next step converts an unbounded error into a caught one.

There is also the unglamorous cost, which tends to be what actually kills a project in month four: a database query answers in milliseconds for a rounding error, while a model call answers in seconds and bills per unit of text, every time, forever.

3.5 The opposite mistake

The test cuts both ways, and teams fail on the other side of it just as expensively.

Somewhere in your organisation, a team maintains forty pattern rules for the phrase "runs small." Then sixty. Then someone adds a second language. Then the person who wrote them leaves, and nobody dares delete any, so the file only grows.

A rule list that grows every week is a model with a worse interface, no evaluation set, and one maintainer who is quietly updating their CV.

If a team is hand-maintaining a taxonomy of human phrasing, they have already built a bad model. Buy a good one.

A useful diagnostic: plot the maintenance curve. If the number of rules needed per month is flat or falling, rules are the right tool and the domain is genuinely enumerable. If it rises without converging, the domain is open-ended and you are paying salary to approximate a model.

3.6 Where the test says yes: customer engagement

Executives often assume this test is an argument against AI in customer-facing work. It is the opposite — customer engagement is where the three shape questions most often return a clean "yes," because customers arrive expressing themselves in language you did not anticipate.

Run the test across a customer journey and the pattern is consistent:

Notice the last two rows sitting side by side in the same conversation. The eligibility decision and the message drafted about it belong to different technologies. Most disappointing "AI customer service" projects fail precisely because they hand both to the same component.

3.7 The bill of materials

Nobody calls their car a "GPS car." The GPS is one component among hundreds; it happens to be the one you look at, so it does not get to name the vehicle.

Yet a four-step automation with one model step routinely gets funded, staffed and reported as "the AI project." That gap between what a system does and what it is called has acquired names — Gartner calls the vendor version agent washing; the US Securities and Exchange Commission calls the investor version AI washing and has fined firms for it.

The discipline that prevents it is mechanical:

1. List every step. Not the goal — the steps. Almost always between four and seven.
2. Label each one: movement, rule, integration, or judgement on unstructured input.
3. Defend every judgement label aloud, one at a time.

If you cannot say in one sentence why a step needs a model, the step does not need a model.

3.8 How to defend the answer

The most common objection to this test is political rather than technical: will saying "most of this is not AI" make my proposal look unambitious?

The experience is the reverse. Stating the line yourself converts you from someone selling a technology into the person in the room doing the arithmetic. A sentence that has never lost an argument:

"Three of these four steps are ordinary engineering, and we built them that way deliberately. One step reads four thousand sentences no rule can catch — that is the step that needed a model. Had we used a model for the other three, you would be paying more for a system that is less reliable."

The proposals that die in the room are the ones where the honest answer is "well, AI is more flexible."

Exercise — Label your portfolio, then build your no-list

Time: 45–60 minutes. Produces the artefact for this module.

Part A — The labelled portfolio. Take every AI project, pilot and vendor demo currently live in your organisation. For each one:

1. Write the steps. Not the goal — the steps.
2. Label each step: movement, rule, integration, or judgement on unstructured input.
3. Write one sentence of justification for every step labelled judgement.
4. Compute the fraction: model steps ÷ total steps.

Part B — The no-list. List the things you have decided not to apply a model to, with one line each on why, referencing which of the five questions decided it.

A yes-list proves you complied with pressure. Only a no-list proves you evaluated. When a board sees four proposals turned down on stated grounds, the fifth — the one you are asking them to fund — stops sounding like enthusiasm and starts sounding like a finding.

What good looks like. Most portfolios come back between 10% and 30% model steps. If yours comes back at 80%, you have either an unusual business or an unexamined deck. If it comes back at 0%, check Question 1 again — you may be under-using models on genuinely unstructured work.

Self-check

1. A process step takes a structured order record and decides whether the order exceeds a credit limit. Which question disqualifies a model, and why?
2. Your team says a step "needs AI because the rules keep changing." Which failure mode might this be, and what evidence would distinguish the two?
3. A vendor proposes a nine-step autonomous agent. Assuming a generous 95% per step, what is the expected end-to-end success rate, and what would you ask them to change?
4. Give an example from your own organisation of the naming problem — where you cannot write the rule because you do not yet know the category.
5. Why is "do not use AI here" almost never the right output of Question 4?

Further reading

• Martin Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google, 2016 — Rule #1: "Don't be afraid to launch a product without machine learning."
• Anthropic, Building Effective Agents, 2024 — on finding the simplest solution and the workflow/agent distinction.
• Gartner, Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027, June 2025 — on agent washing and misapplied use cases.
• MIT Project NANDA, The GenAI Divide: State of AI in Business 2025 — on the share of pilots producing no measurable return.