“The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in and be thoughtful about what you expect, but you also have to expect that there will be a little bit of magic. And that magic is going to be surprising and unique. So your job as a PM is not just to shape what goes into the model, but also to look at what comes out the other side and identify where the model shines.”
Lenny's Podcast: Product | Career | Growth · In-Depth Interviews · November 2026
Context
Asked what people misunderstand about how frontier models are built, Doshi says PMs set goals and capabilities but cannot fully predict the result of a training run. The Harry Potter comparison became the episode's recurring image.
Transcript
…Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much is our audio models. So dialogue, audio understanding, text to speech, translation, all a part of the portfolio of how we build frontier models.
…every week, it feels like. You have a really unique lens in being able to see these models from research to pre-training, post-training launch. What's something that you think people maybe misunderstand about how models are created or something that may surprise people about how models are made?
Yeah, it's a good question. I mean, maybe what I'll say, especially coming from a product management lens, is one thing that I think, you know, when you typically talk about product management for anything, including a model, you talk about what are the goals you have for a model, right? What are the capabilities you want to set? What are the outcomes you want to drive? What goals do you want to hit on a set of evals? And often when I talk to customers or even internal users, you know, they'll ask me, okay, what are the goals that you're setting? And they'll sort of expect that we can really make sure that the model will deliver on those goals in a very traditional product sense. And I think for me, what has often been surprising, but also exciting about model development and being a PM for the model is you should do that and you should have an intentional view, but the model isn't predictable. And so it's kind of like Snape's potions class, right? So you can put all the ingredients in and you should be thoughtful about what you expect, but you also have to expect that there will be a little bit of magic. And that that magic is going to be surprising and that magic is going to be unique. And so your…
It's so interesting that it's just like this code that we have written that we feed data and we need to, like, we don't exactly know what's going to happen and how it works. I recently had an interpretability research person on the podcast from Anthropic, and we talked a lot about this, this idea…
You know, of course, there's brilliant research that goes into these models. And so many aspects of it are predictable in the sense that we are consistently monitoring these runs. We have researchers who are tracking them, who are identifying when to intervene, who are seeing different metrics…
This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies have in common? They're all powered by WorkOS. If you're building for enterprises, you have felt the pain of integrating…
More from Lenny's Podcast: Product | Career | Growth
