Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
CC Chloe Casale On Signals and Threads

“Spending some time thinking about how do I catch those 5%? How do I get this closer to maybe 99%? How do I make sure that when the 5% goes wrong, it's at least easy for someone to notice? Can we have two different prompts parsing the same chat message? Do these things agree? Do they disagree? There's all sorts of ways we can get creative to get a little more confident at catching those 5% that isn't necessarily relying on a person.”

Signals and Threads · Hands-On Engineering Podcasts · November 2026

“Spending some time thinking about how do I catch those 5%? How do I get this closer to maybe 99%? How do I make sure that when the 5% goes wrong, it's at least easy for someone to notice? Can we have two different prompts parsing the same chat message? Do these things agree? Do they disagree? There's all sorts of ways we can get creative to get a little more confident at catching those 5% that isn't necessarily relying on a person.” — Chloe Casale, Signals and Threads

Context

Minsky had floated planting fake errors to test whether traders catch them, which the traders didn't like. Casale prefers checks that don't depend on human vigilance, such as running two independent prompts and flagging when they disagree.

Transcript

Signals and Threads Around 40:21 into the episode
Chloe Casale

pitched it and I think the sales traders were a little less than happy. And I think at the same time, it's like, it's hard to figure out like how much is that actually going to cause people to actually, I don't know, bring up that extra scrutiny level. I think.

Ron Minsky

And you could tell how good they were at it. Yeah. It's tempting. It's

Chloe Casale

tempting. But I think that instead, I think spending some time thinking about like, how do I cache those 5%? How do I get this closer to maybe 99%? How do I make sure that when the 5% goes wrong, it's like at least easy for someone to notice in a way. It's like, can we have two different prompts parsing the same chat message? Do these things agree? Do they disagree? Like, there's all sorts of ways that we can get creative to get ourselves a little bit more confident, at least caching those 5% that isn't necessarily relying on a person. Right.

Ron Minsky

And obviously, you're in a world where the underlying technology is changing really fast. Yeah. It's also maybe not always changing in a way that's good for what we want in the sense that one thing that happens now with models is they spend a lot more time chewing on and thinking about things.…

Chloe Casale

Yeah. So we've been even thinking about, you know, some of our counterparties, like they just can't move on to LLM parsing because they just want their responses too quickly. And it's very hard to guarantee how fast that response is going to come from versus, you know, other counterparties. Like…

Ron Minsky

Right. And there's a question of like, do you give the LLM just like the one chat message or like the history of chat messages where it might in fact be better able to build priors if it sees more stuff, but it's of course going to take longer to do its inference in that world. Have you thought…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Signals and Threads