Tulsee Doshi
On Lenny's Podcast: Product | Career | Growth
“Start with even just five prompts. We often think, oh, I need to build some massive large-scale evaluation or benchmark. But even a mini evaluation, 20 prompts that stress test a set of capabilities, where you can see signal between model A and model B, is already so powerful. Start with yourself as the rater, looking at a prompt and deciding if the response is good.”
Lenny's Podcast: Product | Career | Growth · In-Depth Interviews · November 2026
Context
Lenny asked how someone who has never written an eval should begin. Doshi admits Google is still upskilling its own PMs on evals, and recommends starting tiny with your own judgment as the grader.
Transcript
…it again and rewriting it, right? And so, I think PMs who are constantly keeping an eye out on the environment, on the market landscape, and actually consistently reinterrogating, like, what are we doing and why are we doing it? As opposed to assuming that there was a fixed kind of plan of action.
This is awesome. Okay, so vibes, evals, adaptive. I want to poke at these a little bit. So, with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill of writing evals?
Yeah, and to be honest, we're still figuring that out internally with our PMs as well. And I would say, like, we see some PMs who are stronger at this than others. And so, we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like, start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive large-scale evaluation or benchmark. I think often, like, even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between, you know, your model and another model or model A and model B. I think that already is so powerful. And that's an unlock because it doesn't require you to think as critically about all of the pieces, right? Like, start simple. Start with even just yourself, like, you as the rater, you looking in a prompt, you trying to decide if the result and the response is good, especially when you're talking about these like long, multi-step, multi-turn trajectories. Just even being able to identify how you're thinking about good is really hard…
Awesome. Okay, that is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting, your advice is both actually a really important structured evals or rubrics and also getting good at getting the vibes. So, on that vibes piece…
There isn't necessarily a one-right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this…
This is great advice. So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your…
More from Lenny's Podcast: Product | Career | Growth
