“When they're talking about reliability, they talk, okay, if the LLM can take it up to 90%. And then we say 90%, that means for 60 million phone calls per year, it means an enormous amount of failures that is totally unbearable. So you can really see the difference between people who are thinking about this problem theoretically and people who actually pay the consequences of those mistakes.”
a16z Podcast · Startups & Venture · October 2026
Martone, who runs AI at a company whose agents handle about 5 million calls a month, compares academic papers with production. The host had said even 99% accuracy isn't good enough. Martone says his team judges models on evals run over millions of real conversations, because methods that look the same on a handful of cases diverge at scale.
Transcript
…two things. Like one, what's the one the most impactful? And then two, because we also have to ship at a very fast pace, okay, what's the solutions that can really get us almost as close as possible to the optimal outcome, but in the fastest way possible. And then we have to double down on that.
…traditional products are trying to build these things, especially when there's high risk here. You can't get it wrong. Like it's even if it's 99% accurate, that's not good enough. It has to be 99.999. And they joke that every nine that you add is another hill to climb. It's kind of funny because
like, you know, I still read some papers from academia and there you see really the difference. Like when they're talking about reliability, they talk, okay, if the LLM can take it up to 90%. And then we say 90%, that means for like 60 million phone calls per month per year, it means like an enormous amount of failures that is totally unbearable. So you can really see the difference between people who are like thinking about this problem theoretically and people who actually pay the consequences of those mistakes us, right? We get inundated of like complaints and also like the impact on people's life. And so what we did, which I think is quite exceptional, and to me, at least is what really engages me and really keeps me going, is to build a mixture of expertise, right? So we have at least, we have top academic talents, you know, including like myself, professors of top universities, as well as like top developers. So like you have a team that has all eyes and so they can understand deeply, you know, from coding to like kind of more academic questions. And we try to use all these tools to build evalus, to understand evaluates is kind of the answer to your questions. Like…
no, it's pretty interesting. It's almost like you're applying the scientific method to product design and how you actually build the product.
I 100% agree with you. And I think, you know, I'm a strong believer that the scientific method is kind of one of the best ideas that the human clients have ever generated. Yeah.
I want to talk to you a little bit too about Elise's AI voice model as well, because you specifically have you're able to manage voice latency. I know you guys were working on voice and communication way before even some of the Frontier labs were working on that. But one of the big issues with it…
More from a16z Podcast
