Kendra Vant
I love building.
“It was very common back then to say, I can train this model to be correct 85% of the time. … You never say that to a business colleague. You say I can train this model to be wrong 15% of the time. It's the same number, but it sounds much worse.”
The Product Experience: a Mind the Product podcast · September 2026
Vant is explaining how to test whether an AI product's "reliability layer" is good enough. Her point is to get colleagues thinking early about what happens when the model is wrong, and how the product recovers from it, rather than only about how often it is right.
Transcript
…insights and ideas together, prioritize what's worth building as a team, and create living roadmaps that connect directly to delivery in Jira. So when more ideas are possible than ever, your team can focus on the ones that matter most. Try Jira Product Officer, for free today at atlassian.com
…say, Am I is my reliability layer robust enough? So you've got these questions, but is there something in there that helps me to say, I've asked this and I've gotten some good answers and some bad answers and some things that are a little bit in the gray area. How do I know if I'm robust enough?
I think that absolutely starts with some, even if it's a gut level understanding of how will your users react when it's wrong. So one back in the day, back in the day of traditional machine learning, I would I learned a trick from a colleague. It was very common back then to say, I can train this model to be correct 85% of the time, 92% of the time. And he was right and he taught me, you never say that to a business colleague. You say I can train this model to be wrong 15% of the time. It's the same number, but it sounds much worse when you say that the machine's going to be wrong 15% of the time. And I tried to ask those questions very early in the conversation so that I was pointing out to the folk I was co-creating products with, it's impossible to build any kind of machine, be it generative AI or be it more traditional machine learning, that will be right 100% of the time, it will be wrong some percent of the time. What's going to happen when it's wrong? And so, for instance, I was building a long time ago, I was building a…
I think you actually just demonstrated it really well. The follow-up I wanted to ask was a lot of times the fundamental problem that I'm seeing with these tools is that they don't have the capacity built in or they're not. I'm going to anthropomorphized for a moment. They don't want to say I don't…
Oh, for sure.
And I think that is probably one of the ways of failing in a nice way, of saying being able to say, I'm not sure, I don't know what the right answer for for this is. Is there a way that we can, I think that would actually increase reliability. Quite a bit by being able to declare a fail state or an…