A model demo proves that an output is possible. It does not prove that an automation is safe, repeatable or useful in a real business process.
Pre-launch testing must examine the whole workflow: input, model, tools, business rules, human review and the final action. The result should be calculated by code or verified by people, not graded by the same model that produced it.
Define the accepted business result
Choose the unit that matters: a correctly routed request, an accepted document record or a draft approved without material rewriting. A fluent response is not the unit.
Start with the process and risk map described in the AI implementation roadmap. Record what the system may do, what requires approval and what it must never do.
- required fields and output schema
- quality threshold and prohibited errors
- maximum latency and cost per accepted case
- escalation and human-review conditions
Build a representative golden dataset
A golden dataset is a versioned collection of inputs with agreed expected outcomes. Include normal cases, rare cases, incomplete records, conflicting instructions, multiple languages and deliberately malicious content.
Keep test data separate from production data. Every case needs an identifier, expected outcome, allowed variation and reason for inclusion. When the workflow changes, rerun the same set before adding new examples.
Use deterministic metrics where possible
Validate JSON structure, required fields, identifiers, calculations, links and permissions with code. Use human reviewers for meaning, tone or context that cannot be reduced to a stable rule.
Track acceptance rate, material correction rate, false action rate, escalation rate, latency and cost per accepted case. An average can hide a dangerous category, so split results by language, document type and risk level.
Test failures and hostile inputs
Disconnect an upstream service, return a timeout, send duplicate events, remove a required field and exceed a usage limit. Confirm that the workflow stops safely, preserves evidence and does not repeat an irreversible action.
External files and web pages can carry indirect instructions. Use the controls in the prompt-injection guide: least privilege, isolated tools, output validation and approval before consequential actions.
- prompt injection and conflicting instructions
- invalid or oversized files
- partial API responses and rate limits
- duplicate requests and reordered events
- attempts to access unauthorised objects
Run shadow mode before autonomy
In shadow mode the system receives real traffic but cannot perform the final action. Compare its recommendation with the existing process and review disagreements.
Move from shadow to a limited pilot only when the sample covers normal volume and exceptions. Keep the blast radius small: one team, one process, limited permissions and a reversible output.
Set go, pause and no-go rules in advance
Write thresholds before seeing the results. Otherwise teams redefine success after a weak pilot.
A launch decision should require every critical condition, not a convenient average. Any unauthorised action, corrupted record or failure to stop can be a no-go even when overall accuracy is high.
- Pass deterministic structural and permission tests.
- Meet quality thresholds on each critical category.
- Complete security and failure testing.
- Prove human escalation and manual stop.
- Approve ownership, monitoring and incident response.
Treat evaluation as a production capability
Models, prompts, tools and upstream APIs change. Store the exact configuration with each test run and repeat evaluation after every material change.
The small-business AI governance checklist explains who should own changes, evidence and exceptions. NIST structures this work around Govern, Map, Measure and Manage.
A failure I caught before publication
In my multilingual publishing pipeline, server storage once contained a replacement character in two large Cyrillic HTML fields. The draft looked generally correct, but a byte-level comparison found that a multibyte UTF-8 character had been split during transfer.
The articles remained drafts. I retransmitted the complete bodies with a fixed content length, fetched both records again and compared all 21 fields character by character. Publication continued only after the difference count was zero.
That case changed the acceptance rule: an HTTP success code is not proof of data integrity. The stored object and the public result must match the verified source.
Questions and answers
How many test cases are enough?
There is no universal number. Cover every critical category and continue until new cases stop revealing important failure modes. Risk and diversity matter more than a round count.
Can an LLM evaluate another LLM?
It can support review, but critical acceptance should use deterministic rules, trusted references or accountable human judgement. Do not let one opaque score decide launch.
What is shadow mode?
The system processes real inputs and produces recommendations, but it cannot perform the final business action. This reveals production differences with limited risk.
When must tests be repeated?
After changes to models, prompts, tools, permissions, data sources, schemas or business rules, and on a scheduled basis in production.
Should a pilot use production data?
Use minimised and authorised data. Start with synthetic or de-identified cases, then use controlled real traffic only when the purpose and safeguards justify it.
Operational worksheet
Before approval, put every control into a worksheet with an owner, evidence, threshold and next review date. A statement such as “monitor quality” is not actionable. “Operations owner reviews the material-correction rate each Monday and pauses the workflow above the agreed threshold” can be tested.
Link every control to a business consequence. If a schema failure can corrupt a customer record, specify containment and reconciliation. If a cost alert only indicates harmless seasonal volume, define the context that prevents false escalation.
Keep versions of prompts, schemas, policies, tools and external dependencies. A result without its configuration cannot be reproduced. Record exceptions and their expiry dates so that temporary workarounds do not become permanent invisible policy.
Finally, rehearse the human path. The named owner should be able to find the evidence, stop or restrict the workflow, move cases to fallback and explain the current state without asking the AI system to diagnose itself.