How to evaluate a SaaS vendor AI feature switched on by default

Share Now

A software toggle switch lit blue beside an unopened contract folder, for a SaaS vendor AI feature

How to evaluate a SaaS vendor AI feature switched on by default

Your CRM, helpdesk or accounting vendor flips an AI toggle to on. The feature writes summaries, suggests replies or ranks leads. The question is not whether the demo looks clever. It is how you evaluate a SaaS vendor AI feature before staff treat it as part of the job.

NIST’s AI Risk Management Framework 1.0 is clear that third-party software and data can complicate risk measurement, and that organisations still need to manage risk in the systems they deploy or use. The ICO’s AI and data protection guidance asks for a risk-based approach: assess risks to people’s rights and freedoms, then apply proportionate measures. A default-on vendor feature does not remove either duty.

Five questions before anyone relies on it

  1. What data leaves your tenant, and where is it stored or used for training?
  2. Can you switch the feature off per team, and is off actually off?
  3. What does the vendor document about accuracy, known failure modes and human review?
  4. Who inside your firm is allowed to accept the feature’s output without a check?
  5. What audit trail exists when the feature drafts a customer message or changes a record?

If the vendor cannot answer the first two in writing, keep the feature off for live customer data. The Current Ventures article on when client data should not go to someone else’s model is the practical twin of this checklist.

Pilot shape that fits a small team

Pick one team and one workflow. For example, internal ticket summaries, not outbound customer email. Run for two weeks with mandatory human edit. Log every wrong summary, leaked field or invented commitment. Only then decide whether to widen. Do not measure success as “people clicked the new button”. Measure wrong outputs caught before a customer saw them.

Keep the AI software contract questions nearby when the renewal lands. Default-on features often arrive mid-term through a product update. Renewal is when you can still renegotiate training use, subprocessors and a clear off switch.

Governance without a theatre committee

Write a half-page note: feature name, data in scope, owner, off-switch location, and the human review rule. Store it with your AI working agreement. NIST’s GOVERN function is cross-cutting for a reason. A Slack announcement is not governance. A named owner who can turn the feature off on Friday afternoon is.

When to refuse the default

Refuse when the feature trains on your prompts without a contract path to say no. Refuse when it can message customers without a queue. Refuse when it ranks people for hiring or credit inside a tool that was sold as something else. The ICO’s point about stopping a project when mitigation fails applies to a vendor toggle as much as to a custom build.

Security and privacy checks that fit on one page

Ask for the subprocessor list, the training-data stance, retention for prompts and outputs, residency options, and whether customer content is excluded from model improvement by default. Ask how staff access is logged when someone opens an AI transcript that contains personal data. Ask what happens to drafts if you disable the feature mid-month. Put the answers in the half-page note. If sales engineering will only answer on a call, ask for the same points in an email you can file.

Compare the answers to your existing DPA and to the buying questions in how to connect an AI assistant to tools you already pay for. A feature that can read the whole tenant is closer to a new integration than to a theme change. Treat it that way in change control.

Change control for a quiet Friday release

Vendors love quiet releases. Your change rule should not. Require a named approver before any AI feature that can draft external text or update records is enabled in production. Require a rollback owner. Require a customer-facing pause rule if the pilot error log crosses a threshold you set in advance. None of that needs a PMO. It needs a calendar invite and a shared note.

When the feature touches finance or HR data, route the approval through the same person who owns those systems of record. Marketing can love a summariser. Payroll cannot absorb a silent data leak.

Customer messaging is a special case

Any feature that can email, chat or ticket a customer needs a stricter gate than an internal summariser. Require a human send. Require a tone check against your support macros. Require a pause switch owned by support lead, not only by IT. If the vendor markets “autonomous resolution”, read that as a default to keep off until the error log from assisted mode is boring for a full quarter.

Also watch billing. Some AI features arrive as usage add-ons after the pilot. Ask for unit prices in writing before staff habit forms. A cheap pilot that becomes a meter on every ticket is a budget surprise, not a productivity story.

What good looks like after 60 days

You have an owner note on file. The off switch has been tested once. The error log has fewer than a handful of customer-visible mistakes, all caught in review. Staff can explain in one sentence what the feature is allowed to do. If any of those is missing, keep the feature in assisted mode. Expansion is optional. Clarity is not.

Key takeaways

  • Default-on does not equal approved. Run NIST and ICO risk questions on vendor AI the same way you would on an internal build.
  • Ask where data goes, whether off means off, and who may rely on outputs.
  • Pilot on one internal workflow with mandatory edits before any customer-facing use.
  • Put a half-page owner note next to your AI working agreement.

FAQ

Is a vendor AI feature our responsibility?

Yes. NIST AI RMF 1.0 notes that third-party components complicate risk measurement, and parties still manage risk in systems they deploy or use.

What is the first safe pilot?

One internal workflow, mandatory human edit, a written log of errors, no customer messages.

What if we cannot switch it off?

Treat that as a contract and security issue before staff depend on it. Prefer tools with a real per-team off switch.

Does the ICO ban these features?

No. It expects a risk-based approach and proportionate measures, and it says you may need to stop if risks cannot be sufficiently mitigated.

Where should the decision live?

With a named owner and a half-page note, not only in a vendor webinar recording.

Sources

  • NIST, “Artificial Intelligence Risk Management Framework (AI RMF 1.0)”, 26 January 2023. nvlpubs.nist.gov
  • ICO, “Guidance on AI and data protection: about this guidance”. ico.org.uk

This article is general information, not investment advice. See section 4 of our Terms & Conditions.

This is the kind of decision we advise on. If you are weighing it for your own business, see how we work or start a conversation.

Related Posts