MS Mona Sarantakos On Lenny's Newsletter

“Our system prompt has to be handwritten. You cannot use an AI to write a system prompt, any part of it. It must be handwritten. The evals, the prompt distribution needs to be checked, because you have to make sure this matches what the product reality is. The behavioral spec needs to be handwritten.”

Lenny's Newsletter · Product & Design · November 2026

“Our system prompt has to be handwritten. You cannot use an AI to write a system prompt, any part of it. It must be handwritten. The evals, the prompt distribution needs to be checked, because you have to make sure this matches what the product reality is. The behavioral spec needs to be handwritten.” — Mona Sarantakos, Lenny's Newsletter

Context

Lenny asked how many evals the Muse team has and who writes them. Mona says agents help with a lot of the work. But a person has to write the core specs and check the data by hand.

Transcript

Lenny's Newsletter
Mona Sarantakos

Yeah.

Lenny Rachitsky

…to hear a bit more about the evals. So the way I think about it is you just kind of have these evals that define success for all the core use cases and ways you expect people to use Muse. How many evals do you have roughly? And who's writing these generally? Is it you? Is it PMs? Is it engineers?

Mona Sarantakos

Yeah. We have a ton. I couldn't even count the number of overheads. For sure. Like, so the way that it, so there's major evals, priority evals that we look at. And those tend to be like kind of core. It's like, you know, there's some like, if these don't exist, the product isn't working. Safety is an example of that. Right. So we have like evals for prompt injection. We have evals for general safety for, you know, harm, like all of these kinds of things. So like that's like a whole category. We have a capability, you know, we have capabilities versions of evals. You know, how can like, is it using the browser well? Is it able to, you know, task complete? And then you have like behavioral evals, which is really about the personality of the agent and, you know, how it behaves with the user. Right. And in there, you know, you've got like, is it asking follow-up questions? Is it, you know, how good is it at intent understanding? Right. Can it like figure out what the user is saying? And then you've got your use case evals, like that kind of category, right? Of the priority use cases.…

Lenny Rachitsky

Okay. This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product lifecycle.…

Mona Sarantakos

I mean, this is the number one problem that we have. People are like, why? I already have a chat bot. Like, I don't get it. Why do I need another one? So the most effective lever out there is word of mouth. It's like basically hearing other people's stories. Like that is number one. I was with a…

Lenny Rachitsky

there kind of a non-obvious decision that y'all made that you made in the development of Muse that now feels like a very key part of its success? Well,

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Lenny's Newsletter