Briefing 6 of 8

Selecting AI Tools

If you couldn't say why you picked your AI tool over the other options, did you actually choose it?

A 15-Minute Executive Read

Prepared by 3rd Element Consulting, Mechanicsburg, Pennsylvania

Companion resource: AI Vendor Vetting Checklist

Executive summary

Most businesses don't select an AI tool. Someone finds one they like, starts using it, and it quietly becomes part of a workflow before anyone in leadership makes a decision about it at all. By the time it shows up on anyone's radar, it's already handling client work, and the real question, should we be using this one, gets replaced by a smaller one: how do we stop using it now without disrupting everything built on top of it.

This briefing lays out a short, concrete process for evaluating an AI tool before it gets to that point: what to actually check, who should be in the room, and how to compare two or three options without a spreadsheet full of jargon. It builds directly on the account-type distinctions from Briefing 2, the data-handling rules from Briefing 3, and the ownership and disclosure requirements from Briefing 4.

A good selection process takes a few hours. Unwinding a bad one, after client data has already gone through it, takes considerably longer.

Get this series delivered weekly instead.

What every executive should know

1. Most AI tools arrive by accident, not decision
Here's the specific mechanism: an employee tries a free tool to solve an immediate problem, it works, they keep using it, a coworker sees it and starts too, and within a few months it's load-bearing in a department that never formally approved it. Nobody made a bad decision at any single step. Nobody made a decision at all.
The fix here isn't banning employees from trying new things, but giving them a fast, known path to get a tool evaluated and approved, so trying something new doesn't automatically mean skipping the process.
2. What to actually check before approving a tool
That last item gets skipped more than any other, and it's the one most specific to AI tools right now. A lot of AI vendors are young companies running on venture funding, not established software firms with a decade of customers behind them. A tool that's excellent today can get acquired, shut down, or repriced overnight, and your business is the one that inherits the disruption.
- Business purpose: does this solve a real problem your team already has, using the five-question test from Briefing 5, or is it a tool looking for a use.
- Data access and retention: what information will this tool see, and does the vendor's policy allow them to use it to train their models, the same account-type distinction from Briefing 2.
- Security controls: does it support multifactor authentication, admin-level user management, and activity logging, the software guardrails described in Briefing 4.
- Contract terms: if you cancel next year, can you get your data out in a usable format, and does the vendor commit to deleting it on request.
- Vendor viability: is this a company likely to still exist, and still support this product, in two years.
3. Three ways AI shows up, and each needs a different check
A policy written only with the first category in mind will miss the second one entirely, and most businesses have more AI arriving through embedded features right now than through anyone deliberately signing up for a new tool.
- Standalone AI tools your team logs into directly, like a general-purpose chat assistant: evaluate these the most thoroughly, since they're the easiest for an employee to adopt without anyone noticing.
- AI features built into software you already use, like an assistant inside your email, CRM, or accounting platform: these often arrive automatically in an update, so the check isn't whether to adopt the tool, it's whether to turn the feature on, and under what settings.
- Custom-built AI tools, developed for your business specifically: these need the most scrutiny of all three, since you're also responsible for how the underlying model was built and what it was trained on.
4. Get the right two or three people in the room
A good tool evaluation doesn't need a committee, but it does need more than one person's opinion. At minimum: someone from the department that will actually use the tool day to day, since they'll spot problems a technical reviewer would miss, and whoever owns AI governance under Briefing 4, since they're checking the data and security questions against your actual policy, not against a features list a vendor handed them.
Skipping the second person is how a tool with a compelling demo and a bad data policy gets approved by someone who only ever saw the demo.
5. Compare options with a simple score, not a gut feeling
A basic scorecard, the six checks from this briefing, rated low, medium, or high, laid side by side for each option, turns "which one did we like better" into "which one actually meets our requirements." The tool that wins the demo and the tool that wins the scorecard are not always the same tool, and when they're not, that's worth knowing before you sign anything.

Real business examples

Composite scenarios, illustrative, not specific to any one company.

The tool that became official by default

A 25-person firm noticed, during a routine review, that nearly every account manager was using the same free AI transcription tool for client calls, without anyone ever approving it. It had spread person to person over about six months. The tool turned out to have a reasonable business tier available the whole time. Nobody had looked, because nobody had been assigned to look.

The vendor that disappeared

A marketing agency built part of its content workflow around a promising AI writing startup. Fourteen months later, the startup was acquired by a larger company that discontinued the product with sixty days' notice. The agency had never asked about vendor viability during selection, and had no export process in place. Recreating the workflow on a new tool took three weeks they hadn't planned for.

The feature nobody turned on, and the one nobody turned off

A professional services firm's practice management software rolled out a built-in AI summarization feature automatically, at the same permission level as the rest of the platform, meaning every file in the system was now eligible to be summarized by it. Leadership hadn't evaluated it because nobody realized it counted as a new AI tool rather than a routine software update. Once flagged, it took ten minutes to adjust the settings appropriately. Finding it took four months.

The scorecard that changed the decision

A distribution company was choosing between two AI tools for customer service, one with a slicker demo and a well-known brand, the other less polished but with clearer data handling terms and a longer track record. Running both through a simple scorecard showed the well-known brand scoring lower on data retention and contract terms. Leadership picked the less flashy option, and six months in, it was doing the job without incident.

Decision framework

Six questions to run before approving any new AI tool

Does this tool solve a specific problem someone can name, or is it being adopted because it seems useful in general?
Have you confirmed what this tool does with the data it sees, including whether it's used to train the vendor's models?
Does it support multifactor authentication, admin controls, and activity logging?
Do you know how to get your data out, and have it deleted, if you stop using this tool next year?
Has anyone checked whether this vendor is likely to still be supporting this product in two years?
Has someone outside the requesting department, ideally whoever owns AI governance, reviewed this before approval?

If you answered "no" or "not sure" to two or more of these questions, the tool isn't ready to approve yet, no matter how good the demo was.

Leadership discussion questions

How many AI tools are currently in use at our company that leadership never formally approved?
Do we know which of our existing software platforms have added AI features in the last year, and whether we reviewed them?
If our primary AI vendor shut down tomorrow, how long would it take us to recover, and do we actually know?
Who currently has the authority to say no to a new AI tool, and do they have enough information to use it well?
Are we choosing tools based on a demo, a recommendation, or an actual comparison against our requirements?
What would it take to put a simple, repeatable evaluation process in place before the next tool request comes in?

Action plan

This Week

Check whether any AI features have been added automatically to software you already use in the last year, and whether anyone reviewed them.

This Month

Build a one-page scorecard from the six checks in this briefing, and use it on the next AI tool request that comes in, or on one tool already in use that was never formally evaluated.
Confirm your two or three most-used AI vendors' export and deletion terms, before you need them under pressure.

Next 90 Days

Continue with Briefing 7, Preparing Your Business, to build the readiness this process depends on, and Briefing 8, Your 90-Day AI Plan, to fold tool evaluation into your regular governance rhythm.

This is the sixth in an eight-part series designed to give your leadership team a shared, working understanding of AI, without turning your business into a training exercise.

Companion resource: AI Vendor Vetting Checklist

Get this series delivered weekly instead.

Continue the series

Previous, Briefing 5Business Opportunities Next, Briefing 7Preparing Your Business

Ready to talk through where your business stands? Schedule an IT Environment Review with 3rd Element Consulting.

Schedule an IT Environment Review

Next step