How to run an AI proof of concept that actually tells you something

Plenty of AI pilots end with a nice demo and no decision. That's usually because nobody agreed up front what “success” meant. Here's the approach we use.

1. Pick one real process

Choose something specific, repetitive and measurable, like “processing supplier invoices”, not “using AI in finance”.

2. Measure the baseline

How long does it take today? How many errors are there? What does it cost? Without a baseline you can't tell whether the AI version is better.

3. Use real data

Test with a representative sample of real examples, including the messy ones. AI that only works on tidy examples won't survive contact with your inbox.

4. Define the success criteria

Agree the bar before you start, for example “at least 90% of invoices extracted correctly, with the rest flagged for review”.

5. Keep a person in the loop

Design for review from day one. The goal is usually to have people check exceptions rather than do every item by hand.

6. Decide

At the end you should be able to say clearly: scale it up, change the approach, or stop. All three are good outcomes, because you made the decision on evidence, at low cost.