MS Mona Sarantakos On Lenny's Podcast: Product | Career | Growth

“There's actually been this funny trend recently where our system prompt has to be handwritten. You cannot use an AI to write a system prompt, any part of it. It must be handwritten. … The behavioral spec needs to be handwritten. We use agents a lot, but there's a lot of personal time and effort that goes into this sort of thing.”

Lenny's Podcast: Product | Career | Growth · In-Depth Interviews · November 2026

“There's actually been this funny trend recently where our system prompt has to be handwritten. You cannot use an AI to write a system prompt, any part of it. It must be handwritten. … The behavioral spec needs to be handwritten. We use agents a lot, but there's a lot of personal time and effort that goes into this sort of thing.” — Mona Sarantakos, Lenny's Podcast: Product | Career | Growth

Context

Asked how many evals Muse has and who writes them, Sarantakos says agents help build the evals, but the system prompt and behavioral spec are written by people, because skipping human review always caused problems.

Transcript

Lenny's Podcast: Product | Career | Growth Around 41:40 into the episode
Mona Sarantakos

Yeah.

Lenny Rachitsky

…to hear a bit more about the evals. So the way I think about it is you just kind of have these evals that define success for all the core use cases and ways you expect people to use Muse. How many evals do you have roughly? And who's writing these generally? Is it you? Is it PMs? Is it engineers?

Mona Sarantakos

Yeah. We have a ton. I couldn't even count the number of overheads. For sure. Like, so the way that it, so there's major evals, priority evals that we look at. And those tend to be like kind of core. It's like, you know, there's some like, if these don't exist, the product isn't working. Safety is an example of that. Right. So we have like evals for prompt injection. We have evals for general safety for, you know, harm, like all of these kinds of things. So like that's like a whole category. We have a capability, you know, we have capabilities versions of evals. You know, how can like, is it using the browser well? Is it able to, you know, task complete? And then you have like behavioral evals, which is really about the personality of the agent and, you know, how it behaves with the user. Right. And in there, you know, you've got like, is it asking follow-up questions? Is it, you know, how good is it at intent understanding? Right. Can it like figure out what the user is saying? And then you've got your use case evals, like that kind of category, right? Of the priority use cases. And so, and each of these have…

Lenny Rachitsky

Okay. This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product lifecycle.…

Mona Sarantakos

I mean, this is the number one problem that we have. People are like, why? I already have a chat bot. Like, I don't get it. Why do I need another one? So the most effective lever out there is word of mouth. It's like basically hearing other people's stories. Like that is number one. I was with a…

Lenny Rachitsky

there kind of a non-obvious decision that y'all made that you made in the development of Muse that now feels like a very key part of its success? Well,

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Lenny's Podcast: Product | Career | Growth