How to Run an AI Pilot That Ends in a Decision

An AI pilot is a test with a decision at the end. The decision has three possible outcomes: expand the use, change one thing and test again, or stop. A pilot that ends without one of those leaves the business where it started, minus the time and money the pilot took. Most of what makes a pilot useful happens before anyone logs in.

Pick one task

Choose a task that is:

• Frequent. Done many times a week, so a month of use produces enough examples to judge.
• Measurable. It already leaves a trail of timestamps, counts or a review step.
• Caught in review. A mistake gets found before it reaches a customer or a filing.
• Owned. One manager wants it improved and will run the pilot.
• Allowed. The data involved can go into the tool under its terms.

Good first candidates include drafting replies to routine customer emails, turning meeting notes into CRM updates, and pulling fields from invoices or intake forms. The hardest, most visible process in the business makes a poor first pilot, because too many things change at once to tell what the AI contributed.

Ideas from the people closest to the work tend to make strong candidates. The MIT NANDA report on enterprise AI found that many of the strongest deployments began with power users, and that successful organizations let budget holders and the managers closest to the work surface problems, vet tools and lead rollouts.

Define success before starting

Write the success criteria down and get the person who will make the decision to agree to them before the pilot begins. Anthropic's guidance on evaluating AI applications asks for criteria that are specific, measurable, achievable and relevant, and contrasts a bad criterion, "Safe outputs," with a good one that names a number, a test set and a threshold.

A usable criterion names the measure, today's value, the target, how it will be measured and who measures it. For example: "Median minutes to draft a routine reply falls from [baseline] to [target], with no rise in replies a supervisor has to correct, and at least [number] of [number] pilot users use the tool on most working days."

Set three kinds of criteria:

• A gain. Time, volume or cost improves by a stated amount.
• A floor. Quality does not drop below the baseline.
• An adoption bar. A stated share of the pilot group uses the tool regularly.

A pilot that clears the gain but breaks the floor has not passed.

The MIT report shows why this matters. It counted a project as a success only when it was deployed beyond the pilot phase with measurable KPIs, so pilots with nothing measured could not count, whatever people thought of them. What the MIT 95 percent figure measured goes through that definition and what it means for a business.

Take a baseline

Measure the same things the same way for two to four weeks before the pilot starts: volume, minutes per item, rework or error rate, and cycle time. Use system records over recollection where you can. Without a baseline, the end of the pilot turns into an argument about how things used to be.

Set the scope

Write down what is in and what is out:

• The one task, the team and the systems involved
• The data the tool may use, and the data it may not
• Which outputs a person must check before they go anywhere. In most first pilots, nothing reaches a customer without review.
• The product, plan and settings being tested, so the result can be repeated

NIST's AI Risk Management Framework asks that performance be "demonstrated for conditions similar to deployment setting(s)." Run the pilot on real work in the real systems, with the people who will use the tool afterward, rather than on a demo set.

Choose the users

Pick a mix of experience levels, because the effect can differ sharply between them. In a field study of the staggered introduction of a generative AI assistant across 5,179 customer support agents, productivity measured as issues resolved per hour rose 14% on average, with a 34% improvement for novice and low-skilled workers and minimal impact on experienced and highly skilled workers (NBER Working Paper 31161). A pilot made up only of experts, or only of new hires, would give a lopsided answer.

Keep a comparison group doing the same work without the tool over the same weeks, if the team is large enough. That separates the effect of the AI from a busy month or a new process. The same study measured its effect by giving the tool to different groups at different times.

Train the users before day one, and name one person who answers their questions during the pilot.

Set the length

Fix the end date at the start. The pilot needs to run past the first weeks of learning and cover a normal cycle of the work, including month-end if the task has one.

Speed after the pilot varies too. In the MIT NANDA interviews, top-performing mid-market companies reported an average of 90 days from pilot to full implementation, while large enterprises took nine months or longer.

Review during the pilot and at the end

Check in each week on usage, problems raised, and a sample of outputs reviewed against the quality floor. Admin reports cover usage. Microsoft's Copilot Chat usage report, for example, lists active users, total prompts submitted and active days per user. Fix problems that block use, such as access, training or a weak prompt template, as they come up, and note each change, since it affects the result.

NIST also asks that "Test sets, metrics, and details about the tools used" during testing be documented. Keep the same kind of record for a pilot.

Make the decision

On the end date, compare the results with the criteria written at the start, and choose one:

• Expand. The criteria are met. Plan the next team, the training, the cost at full scale and the owner.
• Adjust and retest. The results are close, or the cause of a miss is clear. Change one thing, such as the task, the prompt template, the tool or the training, and run a shorter second round.
• Stop. The criteria are missed and no single change looks likely to close the gap. Record why. Stopping is a valid result, and it frees the budget and the people for the next candidate task.

Hold the targets where they were set. Moving them after seeing the results turns the pilot back into a matter of opinion.

What to write down

Keep one record per pilot:

With this record, the next pilot starts faster, and a year later there is an answer when someone asks why the business uses the tool.