“When I was playing with one of the early synthesized research products, I asked it for women who had participated in the moon landings and were now in the middle of their careers. And it just fabricated a bunch of 45-year-old women who had walked on the moon.”
Awkward Silences · User Experience · October 2026
Context
Mace on how willing these models are to make things up when instructed the wrong way. He says safeguards are being built in so it's less and less likely now, but it shows why research agents need guardrails and should label what is evidence and what is extrapolation.
Transcript
…you in 18 hours with what we found out and then we'll have a better result. That's the ecosystem I think we ought to be building within our companies because that's what will lead to really high quality decisions that people can trust because the agent's honest about what it knows and doesn't know.
…expert, just on average. And so we would want to have limitations on what they know as opposed to what the models know as well, right? So we want to cap it at what would this kind of person have as their expertise and not go beyond that. So there's both sides of that that I wanted to mention.
Yep, I think you're right. I think the eagerness and willingness of these models to make up stuff if you instruct them in the wrong way is a little bit scary. When I was playing with one of the early synthesized research products, I asked it for women who had participated in the moon landings and were now in the middle of their careers. And it just fabricated a bunch of 45-year-old women who had walked on the moon. And I know safeguards are being built into the system, so that's less and less likely to happen. But it's a reminder of the fundamental mechanism and how you really need to kind of invert that, expose that, and help people not guide it in the wrong direction. And those guardrails are really, really important.
In addition to the guardrails of having the right balance of people who land on the moon. So right, absolutely.
I just want to say any angry Gen A alpha listeners can send their emails to John and anybody that walked on the moon that disagrees with Mike, they can send those emails to Mike, not to me. All right. How do we get started? How does it seem like we can understand this all sort of in an academic…
I mean, I can start with where the professional systems that are built today. So there are things like, I mean, some of the ones that are sort of most popular right now include there's for behavior change. There's one called subconscious AI. Therefore, one that sort of really has a strength in CPG…
More from Awkward Silences
