How to measure added value in AI projects

You might have heard or read that "90% of all AI projects fail", or a comparable number. And that might be right, depending on what you think failure means. But in this case failure means: 'we tried and it didn't work out', and that's not a fail in my view. Especially considering the potential upside of using large scale automation.

While there's a lot of talk about AGI and incoming layoffs, no one really knows what's going to happen. The best thing you can do is try it for yourself and see how well it works for your tasks.

What we can do better is the way in which we try. AI is experimental, which has numerous reasons, one being that it is probabilistic.

One of the ways we can improve the way we try is to take a moment to think about the added value of using AI. Think about the value of the process you're automating before putting considerable time and money into it. Even if at first glance it might seem like the next best model is able to automate your task. Define metrics that matter to your use case, and continuously monitor if you're hitting your target during development and deployment.

I have a simple workflow for determining the added value of using AI. Ofcourse this is simplified, and might not be applicable to your use case.

Let's say we're automating processing invoices.

1. Write out the step-by-step process you're trying to automate. I have found that it works best if you're in a physical room with the domain expert drawing the process on a whiteboard. A lot of value is often in making explicit what's implicit.
2. At each step of this process, define the average time it takes. For instance, opening a PDF, reading the data we need, and then manually inserting that into some other system. Translate that time into money.
3. Take the top three bottlenecks in the process, and find external tools or services available to automate this. For example, try processing a PDF with that tool and see how many times it retrieves the right information.

Hopefully, you end up with a statement like this:

We expect our PDF processing tool to be able to automatically process 80% of documents in 3 seconds, reducing the time spent on PDF processing by 10 hours per week, resulting in $300 per week (~$15K per year) saved.

Some things to consider:

• Make the scope very small. In today's world of AI assistants, it's tempting to build a system that does it all. However, there's plenty of time for next iterations if you solve one thing really well.
• Focus on the easy to do tasks first. AI is good at generalizing, it doesn't work well for edge cases. If humans have a hard time doing the task, so will AI.
• Keep domain experts close, throughout the development and deployment process. I have seen developers working in a silo for months, thinking they could figure it out themselves faster. Only to find out months later that the solution doesn't fit the problem at all.

I'll leave it at that for now.

Every second counts.