Lenny's Newsletter · Product & Design · November 2026
Lenny asked how many evals the Muse team has and who writes them. Mona says agents help with a lot of the work. But a person has to write the core specs and check the data by hand.
Yeah.
…to hear a bit more about the evals. So the way I think about it is you just kind of have these evals that define success for all the core use cases and ways you expect people to use Muse. How many evals do you have roughly? And who's writing these generally? Is it you? Is it PMs? Is it engineers?
Yeah. We have a ton. I couldn't even count the number of overheads. For sure. Like, so the way that it, so there's major evals, priority evals that we look at. And those tend to be like kind of core. It's like, you know, there's some like, if these don't exist, the product isn't working. Safety is an example of that. Right. So we have like evals for prompt injection. We have evals for general safety for, you know, harm, like all of these kinds of things. So like that's like a whole category. We have a capability, you know, we have capabilities versions of evals. You know, how can like, is it using the browser well? Is it able to, you know, task complete? And then you have like behavioral evals, which is really about the personality of the agent and, you know, how it behaves with the user. Right. And in there, you know, you've got like, is it asking follow-up questions? Is it, you know, how good is it at intent understanding? Right. Can it like figure out what the user is saying? And then you've got your use case evals, like that kind of category, right? Of the priority use cases.…
Okay. This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer, are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product lifecycle.…
I mean, this is the number one problem that we have. People are like, why? I already have a chat bot. Like, I don't get it. Why do I need another one? So the most effective lever out there is word of mouth. It's like basically hearing other people's stories. Like that is number one. I was with a…
there kind of a non-obvious decision that y'all made that you made in the development of Muse that now feels like a very key part of its success? Well,