How to evaluate an AI vendor's security: 12 questions for your IT lead

Before an AI vendor touches client data, get written answers on isolation, encryption, what models keep, who approves work, what has been audited, and how you leave. The 12 questions below cover all six. A good vendor answers them plainly. A vague answer is itself an answer.

Why this matters more for AI

AI vendors don’t just store your data. They send it to models, combine it with your business’s knowledge, and act on it. That creates new places for data to leak: model providers that keep prompts, shared systems where one customer’s data sits next to another’s, and automated actions nobody approved.

The FTC Safeguards Rule applies to financial institutions under the Commission’s jurisdiction. Its examples include tax preparation firms, and an accountant in the business of completing income tax returns is a financial institution.

Covered businesses oversee service providers by taking reasonable steps to select and retain providers that can maintain appropriate safeguards, and by requiring those safeguards in the contract.

Isolation

1. How is our data kept apart from your other customers? Look for isolation enforced by the database itself, on top of whatever the application does. In PostgreSQL, row security policies restrict, on a per-user basis, which rows can be returned by normal queries. Application-only separation depends on the vendor’s code adding that filter every time.

2. Can your own staff or services bypass that separation? The table owner is typically not subject to row security policies, and superusers and roles with the BYPASSRLS attribute always bypass the row security system. Ask which accounts can see across customers, and how that access is limited and recorded.

Encryption

3. Is our data encrypted with keys specific to our business? NIST says the proper management of cryptographic keys is essential to the effective use of cryptography for security. We think those keys should be specific to the business, held in a managed key service. NIST does not itself require a separate key per customer.

4. What happens to our keys and data when we leave? Ask for your records as a verified export before the keys are destroyed.

AI models

5. Which AI models see our data, and under what terms? Ask whether the provider retains prompts and outputs, and whether your data is used for training. Microsoft says prompts, responses, and data accessed through Microsoft Graph aren’t used to train foundation models in Microsoft 365 Copilot. That is one product’s term, not a rule for every vendor.

6. Is zero data retention actually in effect, and how is it enforced? Ask whether the control is turned on, and what the system does when it is not. Amazon says setting store=false does not guarantee zero data retention in Bedrock.

If the account is configured for zero data retention and someone invokes a model that requires retention, Amazon Bedrock will block the request and return an error. Ask whether the vendor’s system blocks a model unless those terms are in place. See zero data retention, explained.

7. Is every model call recorded? You should be able to learn which job called which model, when, and with what result.

Control and approvals

8. Does anything reach our clients without a person approving it? NIST says processes for human oversight are defined, assessed, and documented. Our recommendation: agents prepare, and by default a person approves before anything reaches a client.

9. Can approval records be changed after the fact? Ask for approvals that stay tied to exactly what was approved, and for a split between who prepares and who approves.

Independent evidence

10. What’s been independently audited? Ask for a SOC 2 report or its status, and check its scope and date. AICPA and CIMA publish the 2017 Trust Services Criteria for Security, Availability, Processing Integrity, Confidentiality, and Privacy for attestation or consulting engagements to evaluate and report on controls over security, availability, processing integrity, confidentiality, or privacy.

If the vendor does not have the report yet, they should say so plainly and show you their written policies and the controls behind their claims. Treat a certification you cannot verify as a reason to slow down.

11. Where is our data hosted, and how is it backed up? Get the cloud provider, the region, and the backup retention in writing.

Exit and retention

12. Can we set retention and place a legal hold? Your records policy and any litigation hold should override the vendor’s defaults, and deletion should be reviewable before it happens.

How to use the answers

Score each answer as enforced by the system, policy only, or not addressed. “Enforced by the system” is the score you want when a client, insurer, or auditor asks how you chose the vendor. Keep the written answers. They are that evidence.

How Precision AI OS answers

We publish our answers on the security page: database isolation on every table, encryption with keys unique to each business, model calls only under zero-retention routing, with data collection refused on every request, a permanent record of approvals and model calls, and retention and legal hold under your agreement. We haven’t completed a SOC 2 audit yet, and we say so.