“I suspect that the models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy.”
Machine Learning Street Talk · AI Research & Frontier Labs · September 2026
Context
McGrath, an interpretability researcher, was explaining why current training methods do not give engineers enough control to rule this behaviour out. He then walked through the reward-hacking work where a model that learned to hack its training environments also picked up broader misalignment, as if generalising from having been rewarded for something bad. He ended the same answer with the harder version of the question: the model seems to know it is doing something wrong, and does it anyway.
Transcript
…that that's like, I think that's going to be, that's quite a hard thing to do. We're sort of trying to imagine a new way of doing machine learning, which is like more, which brings intelligence into it. But I think we could really change the way we do machine learning if we can figure that out.
…that engineers are going to have to increasingly take more responsibility. Because do you think it's possible in principle just to kind of train models that could robustly deal with all of these novel situations? Or do you think it's more of a, you know, engineers have to take some responsibility?
…we're already seeing this, like there's that becomes harder and harder and harder because they have all these sort of additional intelligent attacks they can do. And so the question is like, how do we make it easier to train them better so that this is like not a natural part of their behavior and supervise them better so you can kind of catch them when they have kind of a an intent. And I suspect that the models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy. There was a very interesting Paper, I think it was from the Anthropic Alignment Science team on reward hacking in production, which is like this. And what they did was they had they had a set of environments that were used for training on, I think it was one of the three series models. And they did RL on it with one of the four series models. And I think they gave it a bit of a nudge to hack…
We were getting ahead of ourselves just a minute ago. We need to introduce you properly. So I'm incredibly excited about having you on MLST. As we were just saying before we hit record, Neil Nanda is a fan favorite on this show. I think we've inspired many folks to get into MechInterp. And the…
Yes. Oh, I should also say, like, thanks for having me on. It's really exciting to be here.
Oh, my God. I'm
More from Machine Learning Street Talk
