“If you optimize for a chain of thought being monitorable, then sometimes the chains of thought are less faithful. And what you end up with is a different objective function that now is prioritizing, I'm going to learn intermediate steps that look sensible, whether or not they are sensible.”
The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis · AI Research & Frontier Labs · October 2026
The host asked why models' written-out reasoning is drifting into a strange dialect of its own. Bigelow points to Goodhart's law: reward only the final answer and nothing keeps the steps human-readable, and rewarding readable steps can get you steps that merely look right. He says he was much more optimistic about monitoring chains of thought a year ago.
Transcript
…to like trying to wrap my head around that and trying to think about like what's going on inside the latent chain of thought and how do we how do we think about that with some of the some of the tools that we have and even just metaphors that we as scientists use for understanding the models.
…explanation, but they certainly have been saying, we're looking into this. It's important. We want to figure out why this is happening. Any intuitions that you would offer that people might want to take inspiration from, chase down, put into the hopper at OpenAI for what the hell is going on there?
Yeah, I think that reasoning kind of presents this really interesting challenge of distribution shift where when we like RL in general, where when you're doing RL, whether it's with like a single model or with sub-agents, there isn't necessarily an incentive to keep things human readable, right? Like the objective is to find the, to have the final, if you're doing like outcome level verification instead of process level verification, then your objective is to basically have a better final answer by whatever means possible. And it kind of reminds me of like things with like language evolution with people where like if you look at a long enough time scale, like human languages change. There isn't really like any driving force that forces the English language to stay exactly as it is today as it was 300 years ago or 600 years ago. And languages do change. I mean, there's like exceptions to this. Like the French government, as I understand, has an institution for sort of keeping the language as it is and has had that for a very long time, which has sort of resisted forces of…
So if
I'm trying to square your comments just now about this quick dramatic evolution of language within chain of thought and within agent agent. Your earlier thoughts on why continual learning won't make things like totally impossible to interpret. Is it just a question of scale? Is it like your model…
just to be clear, when you talk about continual learning, you're talking about the world where we're actually fine-tuning models for each person, right? Actually, like changing their weights. Is that right?
More from The Cognitive Revolution" | AI Builders, Researchers, and Live Player Analysis
