Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
SO Sarah O'Keefe On Pragmatic AI with Matt Stauffer

“Ask the AI a question about something that you are an expert in. Maybe it's knitting, maybe it's video games, I don't know. Nearly always the first answer you get is pretty good, and then ask it for more details. And as you go down that rabbit hole, it'll get worse and inaccurate. Okay. And then ask it a question about something you don't know anything about and do the same thing. And notice that as you go down the rabbit hole, you think it looks okay. And then think about that carefully.”

Pragmatic AI with Matt Stauffer · AI Builders · November 2026

“Ask the AI a question about something that you are an expert in. Maybe it's knitting, maybe it's video games, I don't know. Nearly always the first answer you get is pretty good, and then ask it for more details. And as you go down that rabbit hole, it'll get worse and inaccurate. Okay. And then ask it a question about something you don't know anything about and do the same thing. And notice that as you go down the rabbit hole, you think it looks okay. And then think about that carefully.” — Sarah O'Keefe, Pragmatic AI with Matt Stauffer

Context

In the speed round Matt Stauffer asked whether she had ever cussed out an AI. She said she mostly laughs and takes screenshots, then proposed this experiment for seeing how AI's confidence outruns its accuracy.

Transcript

Pragmatic AI with Matt Stauffer Around 39:14 into the episode
Sarah O'Keefe

…about big AI. So clearly a public-facing chatbot you don't own and you don't have sovereignty over because you've got all these issues. But if you bring it in-house, so you have an enterprise license of one of these things and you bring it in-house, well, then you kind of own it, maybe, right?

Matt Stauffer

Yeah.

Sarah O'Keefe

But there's these really interesting questions about what level of control do you need over your AI systems. And as a big enterprise, are you going to license one of the public systems? Are you going to build your own on an open source or open weight, which are not the same thing system? And then you have to worry about, well, what if that system gets banned in my jurisdiction? So you're in the U.S., you decide to use a Chinese model, it gets banned, or vice versa, right? So if you're a global organization, you have to think pretty carefully about what your AI Fails looks like because you could get yourself in all sorts of trouble, both on the output side and on the input side, if you don't understand the regulatory nightmares that we're dealing with. And I should close this by saying I am not a lawyer and you should definitely talk to your IP lawyers about this.

Matt Stauffer

Yeah. Yeah. I appreciate you saying that. I mean, we build systems for our clients and still I don't think this has come up with any of them yet. So I'm like, I should make sure we dig into that with my lawyer. So I appreciate that. Okay. Are you ready for your speed round?

Sarah O'Keefe

I don't think so, but go ahead.

Matt Stauffer

As ready as you can be. All right. First question is, what is your AI tool? Of choice.

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Pragmatic AI with Matt Stauffer