Questions to Ask an AI Vendor or Consultant

Before signing with an AI vendor or consultant, ask about data, retention, security, integrations, support, exit, pricing and references. A good vendor answers each question in writing, with a link or a document behind the answer. Ask before the contract, and file the answers with it. Most of the questions apply to consultants as well.

NIST's AI Risk Management Framework asks for policies and procedures "to address AI risks and benefits arising from third-party software and data." For a business buying AI, most of that work is asking these questions and keeping the answers.

Data use and model training

• Is customer data used to train or improve the vendor's models, or any other company's models? What is the default on the plan being quoted, and where is that written?
• Which outside model providers and subprocessors handle customer data, and in which regions?
• Can anything a user does change that, such as a feedback button or an opt-in setting?
• How do customers hear about changes to these terms?

Specific answers exist, so expect them. Anthropic states that "By default, we will not use your inputs or outputs from our commercial products" to train its models, and that chats sent as feedback, such as with the thumbs up or down button, can be used for training. Microsoft states that Copilot prompts and responses "aren't used to train foundation LLMs," and also discloses that Anthropic models offered inside Copilot as a subprocessor are currently excluded from the EU Data Boundary. Details like that come out when someone asks where data goes.

Written commitments matter because they can be enforced. The FTC's Office of Technology has said that AI model-as-a-service companies can be liable for breaking promises not to use customer data for secret purposes, "such as to train or update their models," whether "directly or through workarounds." It gives "surreptitiously changing its terms of service or privacy policy" as an example of conduct that risks violating the law.

Retention and deletion

• How long are prompts, outputs, uploaded files and logs kept? Can the customer's admins set that period?
• Is anything kept longer for abuse monitoring or legal reasons?
• When a user deletes something, what happens to backups and logs?

Good answers are specific. OpenAI's API documentation states that abuse monitoring logs are "retained for up to 30 days" by default, and that zero data retention is subject to prior approval by OpenAI. Google lists retention for Gemini in Workspace prompts and responses as "90 days to indefinite, as determined by admins." Some plans include retention controls and others do not. Anthropic's pricing page lists custom data retention controls on its Enterprise plan and not on its Team plan.

Security reports

• Is there a current SOC 2 report, and does its scope include the AI product being sold?
• Can the customer read the report itself, under NDA if needed?
• Is single sign-on supported, on which plan, and at what cost?
• Are security logs included in the standard product?
• Is there a published vulnerability disclosure policy?

A SOC 2 report is a CPA firm's report, under AICPA standards, on a service organization's controls relevant to security, availability, processing integrity, confidentiality or privacy. Read the scope and the exceptions as well as the opinion. How to read a SOC 2 report explains each section, and what a SOC 2 audit asks of AI vendors covers the parts specific to AI.

Certifications attach to specific products. Google states that Gemini "has attained SOC 1/2/3" and several ISO certifications, and also that Gemini Notebook and Gemini in Chrome do not currently support ISO, SOC or FedRAMP compliance. Ask which products the report covers.

CISA's Secure by Demand guide gives software buyers questions of the same kind, including whether a manufacturer supports "standards-based single sign-on (SSO) for customers at no additional cost," and it says manufacturers "should make available security logs to customers in the baseline version of the product."

Integrations

• Which systems does the product connect to today, and is each connection read-only or read and write?
• Whose permissions does each connection use, the user's or a shared service account's?
• Who at the customer can turn a connector on?
• When the other system updates and the connection stops working, who fixes it?

For assistants that search company files, ask about permissions first. Microsoft states that Copilot "only surfaces organizational data to which individual users have at least view permissions." That is the right design, and it also means a file shared too widely can turn up for everyone it is shared with. Check sharing settings before connecting anything.

Support and model changes

• Who answers when something breaks, in what hours, and how fast?
• Is there a named contact for the account?
• How are customers told when the underlying model is updated or retired, and are their workflows retested when it is?

A vendor that builds its product on another company's model should explain how it handles that model being updated or retired, and how much notice its customers get.

Exit and data export

• How does a customer get its data out, in what format, and what is included?
• Are uploaded files, custom instructions and workflow settings included, or only chat history?
• After a customer leaves, when is its data deleted, and is there written confirmation?

Exports differ in what they include. Anthropic's help center says that on Team and Enterprise plans, members can export their own chats once an owner turns the setting on, that this export "doesn't include the contents of files you uploaded," and that Primary Owners export the organization's data from organization settings. Know what you would leave with before you start.

Pricing model

• Is the price per seat, per use, or both? Is there a seat minimum or an annual commitment?
• What happens when usage goes over the plan, and can spend be capped?
• Do needed features, such as audit logs or retention controls, sit on a higher tier?
• How much can the price change at renewal?
• For a consultant: fixed fee or hourly, what is included, and what triggers a change order?

Published plans show how much the structure varies. Anthropic's Enterprise plan is "US$20/seat/month, billed annually, 20 seat minimum," with usage billed separately at API rates, and it includes audit logs and custom retention controls that the Team plan lists as "No." Ask for a written estimate at your expected usage.

References

Ask for two or three customers of similar size, doing similar work, who have used the product or the consultant for more than a few months. Questions for them: How long did it take to get working? What broke? How did support respond? Would you buy it again?

How a consultant should scope work

A consultant worth hiring learns how the work runs before quoting. A sound scope names:

• The specific workflows in scope, and the ones out of scope
• The current measures for each workflow, taken before any change, and the target
• The systems touched and the data used, with who approves access
• What the business will own at the end: documents, configurations, prompts and training materials
• Who on the client side is needed, and for how many hours
• How and when results are reviewed, and what happens if targets are missed
• Whether the consultant earns referral fees or resale margin from any vendor they recommend

Referral fees are not a problem in themselves, as long as you know about them when you weigh a recommendation.

Answers that need a follow-up

Some answers should send you back with more questions:

• No written answer on whether your data trains models
• "We're SOC 2 compliant" with no report, or a report that does not cover the product you are buying
• A price you cannot estimate for your expected usage
• No way to export your data, or an export that leaves out what you would need
• A demo only on the vendor's own data, and reluctance to test on your examples
• Accuracy claims with no description of how they were measured
• A consultant who quotes before learning the workflows, or a plan with a long build before any measured test
• Pressure to sign a multi-year contract before a pilot

A short pilot on your own work, with success criteria agreed before it starts, settles most of what remains.