A successful AI demo proves that a model can produce an interesting output. A production system must prove much more: repeatability, ownership, security, cost control and recovery from failure.
The UK AI Playbook covers the full path from deciding whether AI is appropriate to building, buying and using it safely. The useful lesson is that deployment is a sequence of evidence gates, not one launch date.
Stage 1: discovery and baseline
Define one process and one accountable owner. The scoring method in How to Choose the Right AI Use Case helps reject projects that are broad, irreversible or impossible to measure.
Record current volume, cycle time, human effort, correction rate, exception rate and cost. Without a baseline, a pilot can only demonstrate activity.
- Deliverable: one-page use-case brief.
- Gate: value, owner, baseline and stop condition are agreed.
Stage 2: data and permission map
List every source, field, document, retention rule and role. Separate data required for the task from data that is merely available.
Create a representative evaluation set with normal cases, difficult cases and known exclusions. Keep it separate from prompt development examples.
- Deliverable: data map and evaluation dataset.
- Gate: access is lawful, minimal, current and technically available.
Stage 3: constrained prototype
Build the smallest path that can test the hypothesis. Use read-only access or sample files and produce recommendations or drafts instead of external actions.
Choose a chatbot, workflow or agent only after the path is understood; this architecture comparison explains the trade-offs.
- Deliverable: working prototype with logs.
- Gate: the system completes the core task on representative examples.
Stage 4: pilot with real work
Run real cases in shadow or draft mode. Review every output, record failure categories and measure the additional work created by corrections.
Do not tune only for average quality. Track serious failures, missing evidence, wrong tool use and cases where the system should have escalated.
- Deliverable: pilot report against the baseline.
- Gate: acceptance, exception, safety and cost thresholds are met.
Stage 5: evaluation and assurance
UK government guidance on AI testing and assurance describes a modular approach that selects testing activities according to the system and use case.
Evaluate model output, the surrounding workflow and human review separately. Include regression tests so a prompt, model or data change cannot silently remove an important behavior.
- Deliverable: versioned evaluation suite and acceptance criteria.
- Gate: results are reproducible and known limitations are documented.
Stage 6: production integration
Move authentication, permissions, validation, calculations, idempotency and writes into deterministic services. Give the model only the tools and fields required for the task.
Add queues, retries, timeouts, duplicate protection and an auditable link between each input, model output, human decision and external action.
- Deliverable: production architecture and incident runbook.
- Gate: security review, load test and rollback test are complete.
Stage 7: limited launch
Release to a small user group, one team or a percentage of cases. Keep manual fallback and compare production results with the pilot dataset.
The controlled-pilot sequence in AI Automation for Small Business remains useful even when the project is technically complex.
- Deliverable: launch report and owner sign-off.
- Gate: production quality and cost remain inside limits.
Stage 8: monitoring and change control
NIST AI 800-4 organizes post-deployment monitoring into categories and documents the practical gaps that appear when AI meets changing real-world contexts.
Monitor input drift, output quality, tool errors, latency, cost, human overrides, incidents and user behavior. Every model, prompt, retrieval source and policy change should be versioned and re-evaluated.
- Deliverable: dashboard, alerts, review schedule and change log.
- Gate: the system has an active owner, not just a running endpoint.
Use four decisions at every gate
NIST AI RMF frames risk work through Govern, Map, Measure and Manage. At each stage, decide whether to proceed, revise, narrow or stop.
Stopping a weak case after discovery is a successful decision. Sending it to production because time has already been spent is not.
A production transition from my work
The blog API pipeline began as a manual test with one draft. I first verified the real endpoint, required fields and image-upload behavior without publishing.
Then I added structured payloads, full PUT updates, language-specific links and an eight-point live-page check. Only after draft creation, read-back and explicit authority did the status change to published.
The important transition was not a smarter prompt. It was adding source-of-truth state, deterministic checks, a reversible draft stage and post-publication verification.
Questions and answers
How long should an AI pilot run?
Define a sufficient number of representative cases and exceptions, not only a calendar duration.
When is a prototype ready for production?
When the full system meets documented quality, security, cost, rollback and ownership gates on real work.
What should be monitored after launch?
Inputs, quality, tool calls, exceptions, human overrides, latency, cost, incidents and changes to models or data.
Can a vendor demo replace a pilot?
No. A demo does not test your data, permissions, integrations, users, exceptions or failure costs.
Who owns a production AI system?
A named business owner owns outcomes, while technical, security, legal and operational roles own their controls.