Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
SO Sarah O'Keefe On Pragmatic AI with Matt Stauffer

“The research says that it takes as few as 50 or 100 documents to poison a public-facing AI with bad information. So the implication is I should be able to get good information in there. We'll turn that to good. But it's a lot of work and it's very hard to control.”

Pragmatic AI with Matt Stauffer · AI Builders · November 2026

“The research says that it takes as few as 50 or 100 documents to poison a public-facing AI with bad information. So the implication is I should be able to get good information in there. We'll turn that to good. But it's a lot of work and it's very hard to control.” — Sarah O'Keefe, Pragmatic AI with Matt Stauffer

Context

Matt Stauffer asked whether content owners have any control over public chatbots such as ChatGPT or Claude. She says influencing them is surprisingly similar to SEO, and cites research on data poisoning as a sign of how little it takes to shift what they say.

Transcript

Pragmatic AI with Matt Stauffer Around 28:05 into the episode
Sarah O'Keefe

It's just always three. Yeah. It's always three. And you can go in there and you can look at the code and you can say, oh, crap, and you can fix it.

Matt Stauffer

Yeah.

Sarah O'Keefe

Probabilistic, which is AI, machine learning, and all that, you know, broadly, that whole bucket of things, one plus one is sometimes a different number. Usually it's two, but maybe not. And so at the end of the day, the question is: how deterministic does your content need to be? How predictable do you need it to be? And what are the use cases where predictability is critical? And what are the use cases where having it be more probabilistic and giving you the average of what's in the database is acceptable? That's the determination that you have to make. And so there's a lot of really fascinating questions about how you deal with this and what you do and what are your standards. Now, I think the brutal reality is that for product and technical content, generally the standard has not been that high. Like a lot of this content's not very good. And so just getting it to average feels like an improvement potentially. So here we are.

Matt Stauffer

Yeah. So I'm going to, that was really, really, really helpful context. I'm going to re-ask one piece of my question because I'm still curious. Do you feel like your clients and the authors or the readers of this book have any control over how, I don't know what the right phrase is, but like…

Sarah O'Keefe

So first, I mean, it's obviously true that if you build your own internal private processor, private chatbot, that you can control that. You can tell it what information it has access to. You can exclude things. You can tell it not to ask entertain questions like, what is the weather tomorrow? And…

Matt Stauffer

Yeah.

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Pragmatic AI with Matt Stauffer