AI Governance / Procurement

The Questions Your AI Vendor Is Hoping You Don't Ask After the Demo

The demo went well. The interface was polished, the answers were quick and the sample data behaved perfectly.

Atika Mehta10 min read

Key takeaways

  1. 1Why the Demo Is the Least Informative Part of Buying AI
  2. 2Why Smart Buyers Still Skip the Hard Questions
  3. 3The Risks That Sit Behind a Good Demo
  4. 4The Questions to Ask Before You Sign
  5. 5Building a Repeatable AI Vendor Due Diligence Process
  6. 6Frequently Asked Questions

“A demonstration shows what a product can do on its best day, with prepared data and a patient presenter. Procurement exists to find out what it does on every other day.”

Adapted from long-standing enterprise software procurement practice

The demo went well. The interface was polished, the answers were quick and the sample data behaved perfectly. That is precisely why it is a poor basis for a buying decision. A demo shows a product at its best, and the questions that decide an AI purchase — about your data, accuracy, accountability and exit — are rarely answered by it. Many vendors will answer them honestly if asked. The problem is that buyers often do not ask. This guide is for business, security and technology leaders who want a practical set of questions to put to any AI vendor after the demo and before the contract.

American Meteorological Society 94th Annual Meeting (14272542629)
Prospire
Illustrative. What a demo shows versus what you need to know.

Why the Demo Is the Least Informative Part of Buying AI

Software demos have always been staged, but AI widens the gap between demo and reality. Conventional software is largely deterministic: if it works in the demo, it will usually work in production. AI is not. Its output depends on the data it receives, how people phrase requests, the model version underneath and the context around each interaction. A system that handles a curated example flawlessly may behave very differently on your messy, real-world inputs.

Demos also concentrate on capability, because capability sells. They rarely cover operation: what happens when the system is wrong, when your data is sensitive, when the vendor changes the underlying model, or when you want to leave. Those are the factors that decide whether a purchase delivers lasting value.

What a demo cannot tell you

  • Performance on your data. Prepared examples do not represent your documents, customers or edge cases.
  • Behaviour over time. Models are updated, and a change on the vendor side can alter outputs without any change on yours.
  • Behaviour when wrong. Demos show successes. You need to know how failures are detected, reported and corrected.
  • Total cost. Usage-based pricing, integration work and support tiers seldom feature in a live presentation.
  • Data handling. Where your information goes, who can see it and how long it persists are invisible on screen.

Why Smart Buyers Still Skip the Hard Questions

It is tempting to blame carelessness. In practice the reasons are more human, and recognising them is the first step to getting past them.

The pressures that suppress good questions

  • Urgency. Leadership wants an AI win quickly, and each extra question feels like delay.
  • Technical intimidation. Business buyers worry that asking about model behaviour or data architecture will expose gaps in their knowledge.
  • Logo comfort. A well-known customer name feels like safety, even though that customer's use case may be nothing like yours.
  • Fragmented ownership. Procurement, security, legal and the business sponsor each assume another function is checking the details.
  • Sales momentum. Time-limited discounts and pilots that quietly become production deployments both reduce scrutiny.

None of these is a failure of intelligence. They are structural, so the remedy is structural too. Give every evaluation a shared, agreed list of questions, so that asking is routine rather than adversarial. When every vendor receives the same list, nobody is singled out, and good vendors welcome the chance to stand apart.

Evaluating an AI vendor and unsure what to ask?Talk to our AI strategy specialists ›

The Risks That Sit Behind a Good Demo

Skipping diligence rarely causes an immediate failure. It creates latent risk that surfaces months later, when switching is hardest and the stakes are highest.

What tends to surface after signature

  • Data exposure. If sensitive information is retained, passed to sub-processors or used to improve the vendor's models, you may have created a compliance issue you never intended. This matters particularly in healthcare and life sciences and financial services.
  • Unmeasured accuracy. Without agreed measures of quality, nobody can tell whether the system is performing, drifting or quietly producing errors.
  • Lock-in. Proprietary formats, embedded workflows and unclear data portability can make leaving prohibitively expensive.
  • Opaque change. If the vendor swaps the model or alters prompts, behaviour shifts without your approval.
  • Accountability gaps. Contracts that disclaim all responsibility for outputs leave you holding the liability for decisions you did not fully control.
  • Cost escalation. Usage-based pricing can grow far faster than planned once adoption spreads.

The Questions to Ask Before You Sign

The questions below are grouped by theme. You do not need technical expertise to ask them. You need only to insist on clear, documented answers, and to notice when you do not get one.

Data and security

  • Where is our data processed and stored,. and in which jurisdictions? Can we restrict this?
  • Is our data used to train or improve models,. for us or for other customers? Can that be excluded in the contract?
  • Who are your sub-processors,. and how are we told when they change?
  • How do you isolate customers from one another,. and can you show us how single sign-on, role-based access and audit logging work in practice?
  • Can we see evidence of an independent security review,. and how are fixes released without disrupting us?

Performance and change

  • How do we measure accuracy on our own data,. and will you support a structured trial against agreed criteria before we commit?
  • Which model powers the product,. how are we notified before it changes, and can we pin a version?
  • What happens when the system is wrong,. and how are errors detected, escalated and corrected?

Governance and exit

  • What audit trail do we get. of who used the system, with which inputs, and what it produced?
  • What is your liability position. for outputs, and what indemnities do you offer?
  • How do we leave?. In what format can we export our data and configurations, and at what cost?
  • What does total cost look like. at a multiple of our expected usage?

Building a Repeatable AI Vendor Due Diligence Process

A list of questions is a good start. A process makes it repeatable, so that quality of diligence does not depend on who happens to run the evaluation.

A process you can adopt

  1. Tier the use case by risk. Classify each proposed use by data sensitivity and the impact of an error. A tool that summarises public content needs less scrutiny than one that touches customer records.
  2. Standardise the questionnaire. Use the same core questions for every vendor, with extra sections for higher tiers.
  3. Run a structured trial. Test on your own representative data against success criteria agreed in advance, not on the vendor's samples.
  4. Review jointly. Bring security, legal and the business owner into one session so concerns surface early and no function assumes another has them covered.
  5. Contract for what you learned. Reflect the answers in the agreement: data terms, change notification, audit rights, service levels and exit provisions.
  6. Monitor after go-live. Diligence does not end at signature. Track quality, usage, cost and vendor changes continuously.

This process sits inside a wider AI strategy, in which vendors are chosen deliberately against business priorities rather than accumulated opportunistically. It also helps to have a view of how AI is used once deployed, and to know where building your own autonomous AI agents makes more sense than buying. Prospiretech is an OpenAI Select Partner and an Anthropic Partner, but we advise on the decision, not only the delivery. If you want to see what a vendor prepared for scrutiny looks like, read how an enterprise SaaS company in the USA made a multi-tenant platform enterprise-ready from the ground up.

Want a due diligence checklist tailored to your sector?Book a complimentary 30-minute consultation ›
Case Study · Cloud & DevOps Engineering

Making a Multi-Tenant SaaS Platform Enterprise-Ready from the Ground Up

Everything above is a question a buyer should put to a vendor. This case shows what it looks like from the other side. An enterprise SaaS company in the USA that builds AI governance tools needed its multi-tenant platform to withstand the security questionnaires, SSO requirements and data isolation standards that enterprise procurement teams apply. As development partner, Prospiretech containerised the stack with Docker, automated testing and deployment through GitHub Actions with automated rollback, enforced tenant isolation at the data layer, and designed authentication to be Okta SSO/SAML-ready. The aim was to build for scrutiny from day one, not retrofit it after a buyer asked. For buyers, it is a useful picture of what a vendor prepared for scrutiny can show.

  • The platform passed enterprise security review without any architectural changes post-delivery
  • New enterprise tenants can be onboarded without infrastructure changes, because isolation is built in
  • Releases are fully automated, with no deployment window to coordinate
  • Enterprise customers can connect their own identity provider without additional engineering
Read the full case study ›

Frequently Asked Questions

Why is a demo not enough to evaluate an AI product?

A demo shows a product at its best, on prepared data, with an experienced presenter. AI outputs vary with inputs, context and model version, so performance on your real data can differ considerably. Demos also rarely cover data handling, error management, change control, liability or exit terms. Those operational questions determine long-term value and risk, and they need to be answered through structured trials, documentation and contract terms. Treat the demo as an introduction, not evidence.

What are the most important questions to ask an AI vendor?

Focus on four areas: how your data is stored and processed, and whether it trains models; how accuracy is measured on your own data; how model changes are communicated and controlled; and what happens at exit, including data portability. Add audit trails and liability. If you can only ask a few, ask the ones where a poor answer would be hardest to fix after signing. Write the answers down and keep them with the contract file.

How do we test an AI product fairly before buying?

Agree success criteria in advance, then test on representative samples of your own data, including difficult and unusual cases. Involve the people who will use the product and compare results with your current process. Keep the trial separate from the vendor's own examples and record what you find. A structured trial also gives you stronger negotiating evidence than impressions from a presentation. Include a few cases the vendor would prefer not to show.

What if the vendor will not share details about its models or data practices?

Some detail may be commercially protected, but a vendor should be able to explain data handling, retention, sub-processors and change notification in writing and in the contract. Persistent vagueness is itself informative. Consider limiting the use case to lower-risk data, adding technical controls on your side, or choosing a more transparent alternative. Ask for the position in writing, so that it can be compared across suppliers. Document the reasons for your decision either way.

Does due diligence slow down AI adoption?

It adds time upfront, but usually saves more by preventing rework, compliance problems and costly vendor changes. A standard questionnaire and risk tiers keep the process proportionate, with light checks for low-risk uses and deeper review for sensitive ones. Over time a repeatable process makes buying faster, because teams know what is expected and vendors know what to provide. Start with a light version and refine it with each evaluation.

Share this article

Ask the questions before you sign, not after

A disciplined evaluation protects your data, your budget and your freedom to change course. Prospiretech helps you design a due diligence process and an AI strategy that match the risk of each use case, whether you buy, build or both.

All articles

Your AI journey starts with one conversation.