Field Notes / 01

The pilot works.
The company hasn’t changed.

A convincing demonstration answers a technical question. It leaves an organisational one open: what needs to change around the model?

01The task02The workflow03The review04The owner

The screen looks promising. A document becomes a summary. A question becomes an answer. The demonstration ends with a familiar instruction: let’s roll this out.

There is still a decision to make. A pilot shows what can happen under a particular set of conditions. Everyday work asks what happens when the information is incomplete, the request is unusual or the person who built the demonstration is unavailable.

The distance between those two settings deserves as much attention as the model itself.

Begin with the work that surrounds the answer.

Consider a hypothetical proposal-writing pilot. It produces a polished first draft from a customer brief. That is a useful capability. But a proposal also contains commitments: a price, a delivery assumption, a description of what the business can actually provide.

Someone must check those commitments. Someone must resolve missing information. Someone must decide which source is current. The useful unit of analysis is the whole path from the request to an approved proposal.

Drawing that path can expose a question a model comparison cannot answer: are we removing effort, moving it to another person or introducing a new kind of review?

Give the exceptions an owner.

The ordinary case makes a demonstration easy to follow. The awkward case reveals how the process is organised.

What should happen when the source documents disagree? When a customer asks for something outside the offer? When the system produces an answer that reads well but cannot be supported? A person needs both the responsibility and the authority to resolve the issue.

‘Human oversight’ becomes meaningful when the reviewer knows what to check, can refuse the output and has a route for getting the underlying problem fixed. Adding a name to the last box of a diagram is not enough.

The unit of change is the workflow. The answer is only one part of it.

Measure the result after review.

The time to generate an answer is only one part of the task. Evaluation should also account for checking, correction, approval and the consequences of an unresolved error.

For the hypothetical proposal, a practical comparison could include elapsed time to approval, the amount of rework and whether the final document accurately reflects the agreed offer. The relevant measures depend on the work; there is no universal score that settles the business decision.

Test the ordinary cases and the exceptions you can reasonably anticipate. Define in advance what would support expansion, what would trigger a redesign and what would make you stop.

Make the next decision small enough to inspect.

A useful next step might be one workflow with a named owner, a limited group of users and explicit review requirements. It need not be a company-wide programme.

The NIST AI Risk Management Framework treats trustworthiness as a consideration across the design, development, use and evaluation of AI systems. That is a useful reminder to look beyond the demonstration itself.

Our view is straightforward: keep the technical ambition, and make the surrounding decisions explicit. A pilot has done its job when it helps the organisation choose its next step with better evidence.

Which workflow are you deciding about?

Explore AI strategy & implementation ↗