A polished demo answers the vendor's best question with its best configuration. Procurement must answer a different question: will this system remain useful, secure and supportable inside your process?
Ask for evidence that maps to your use case. Treat claims without a method, sample or contract commitment as marketing.
Start with the job and the risk
Define the business outcome, users, data, decisions, integrations and price of error before comparing products. The same service can be low risk for drafting public copy and high risk for customer decisions.
Use the AI use-case selection framework to set scope and acceptance criteria.
Map data flows and legal roles
Ask what data is collected, where it is processed and stored, whether it is used for training, who can access it, retention, deletion, backup and export procedures.
Identify controller, processor and subprocessor roles with qualified legal advice for the relevant jurisdiction. ICO's AI audit toolkit calls for due diligence before procuring AI systems and for clarity around contracts and third parties.
- data categories and purpose
- processing and storage regions
- training and product-improvement use
- subprocessors and change notification
- retention, deletion and customer export
Demand security evidence
Review identity controls, SSO, MFA, least privilege, encryption, tenant isolation, vulnerability handling, incident notification and independent assurance. Ask what customers must configure themselves.
CISA's Secure by Demand guide is designed to help software buyers include security in procurement discussions. Evidence matters more than a badge without scope.
Test quality on your cases
Request the vendor's evaluation method, but run your own representative dataset. Measure accepted outcomes, serious errors, correction time, escalation and stability across languages or document types.
Follow the pre-launch testing framework. A vendor benchmark cannot replace your business acceptance test.
Inspect integration and operational controls
Check API versioning, rate limits, export formats, webhooks, logs, regional availability, status history, support process and rollback options. Confirm what happens when the model or product changes.
Service levels should cover what matters to the workflow, not only platform uptime. A service can be available while quality or latency makes the process unusable.
Price total ownership and lock-in
Model subscription, usage, integration, data work, monitoring, review, support and migration. Test whether the company can export data, prompts, evaluations, audit records and configurations in usable formats.
Compare the result with the build-versus-buy decision. The lowest starting price can carry the highest exit cost.
Put the exit plan in writing
Define termination assistance, export timing and format, deletion confirmation, continuity window, ownership of custom work and the response if the vendor closes or changes terms.
Pilot contracts should preserve the ability to stop. Do not let a successful demo become an irreversible architecture decision.
How I separate SaaS from the control layer
In my workflow I use ready-made services for task state and team notifications, but keep the publishing contract and verification logic in my own API layer.
That split lets me replace a communication tool without changing article data, and change content generation without giving a model direct control over publication. The source record, approval gates and public-page checks remain under one explicit workflow.
This practical boundary is part of vendor evaluation: I buy replaceable capabilities and retain the control that protects data integrity.
Questions and answers
Which question should be asked first?
What exact business outcome, data and decision will the product handle? Without that, security and quality answers have no context.
Are certifications enough?
No. Check scope, date, exclusions and whether the control applies to the service and region you will use.
Should vendor benchmarks be trusted?
They are useful evidence about a defined test, not proof for your workflow. Run your own cases.
What creates vendor lock-in?
Proprietary data formats, unavailable logs, embedded workflows, nonportable evaluations, contract limits and high migration effort.
When should procurement stop?
When the vendor cannot explain material data use, security responsibilities, quality evidence, incident handling or a workable exit.
Operational worksheet
Before approval, put every control into a worksheet with an owner, evidence, threshold and next review date. A statement such as “monitor quality” is not actionable. “Operations owner reviews the material-correction rate each Monday and pauses the workflow above the agreed threshold” can be tested.
Link every control to a business consequence. If a schema failure can corrupt a customer record, specify containment and reconciliation. If a cost alert only indicates harmless seasonal volume, define the context that prevents false escalation.
Keep versions of prompts, schemas, policies, tools and external dependencies. A result without its configuration cannot be reproduced. Record exceptions and their expiry dates so that temporary workarounds do not become permanent invisible policy.
Finally, rehearse the human path. The named owner should be able to find the evidence, stop or restrict the workflow, move cases to fallback and explain the current state without asking the AI system to diagnose itself.