From ML engineering to AI engineering

I graduated from a masters in Data Science in 2021. This means I've been active in machine learning some time before the genAI boom happened. While the fundamentals have stayed the same, some things have changed.

I am currently reading Chip Huyen's AI Engineering. It's the perfect book about AI engineering if you come from a more predictive, or traditional, machine learning background. The book put some things into words that I have felt, but I wasn't conscious of.

What has stayed the same

Two fundamentals of developing AI systems still apply. The first being experimentation.

We need to experiment because we can't tell beforehand how well something will work for our problem. The best ROI comes from doing, rather than talking about it. Which actually holds for more things in life than just machine learning, but that's another story.

Let's say you've tried ChatGPT for extracting data from an email. At first glance it works well. Before you dive in and start implementing it for all your emails, you run a little experiment. It's good to determine the added value beforehand what you expect. You continue developing your system while it aligns with the goal you set.

The second fundamental is evaluation. What you evaluate has changed, but the way you evaluate is still more or less the same.

In predictive machine learning as well as generative AI, you keep domain experts as close as possible to determine whether its the correct output. For generative AI that might mean going over historical chats to see what goes wrong. Hamel Husain has some great content online about evals for gen AI. For predictive machine learning, you typically go over false positives or false negatives and find out why the model was wrong. Through these analyses you improve your AI system.

What has changed

In predictive machine learning, you spend time on training a model. In Gen AI, foundation models are ready for use. You spend more time in the application development layer and less in the model development. In Gen AI projects I use my knowledge of statistics and probability a lot less than I did before.

More opinionated, but what's also changed is the speed of certain phases in development. In Gen AI, the time to go from 0 to 70% is quite fast. You can impress management with a cool demo in a few hours. However, the time from 70 to 100% is way longer. This has to do with the open ended nature of generative AI. In predictive machine learning, we always get the same answer for each input. In Gen AI, we can get ten different answers for the same input.

What has also changed for me is how non-technical people look at AI. During my first project after graduation, I had to explain to people what AI could do. Now I have to tell people what AI can't do. There is a narrow vision of what AI is. That vision is formed by people's incidental use of ChatGPT type tools. Especially now that software providers like Salesforce and Azure are pushing agents. People get discouraged fast if the results are bad.

Innovation happens at the speed of trust.

Meanwhile, the way to go is forward, which means you just gotta try and see how well this AI thing works for you.