Dwarkesh Patel
Hosts the Dwarkesh Podcast, a long-form interview show on AI progress, timelines, and the people building frontier models, along with economists and historians.
“I wrote a blog post a few months ago about how RL, at least policy gradient RL, is even more inefficient than you might think. The inefficiency one thinks about naively is that you have to roll out a whole trajectory to get any learning signal at all — instead of completing the next word in a sentence, the model has to do two days' worth of work to figure out if it even did the project correctly. If you're doing this through RL, the model would try 'the sky is halcyon' — nope, wrong. 'The sky is told' — nope, wrong. This is a totally untrained model, so you'd have to do this on the order of 100,000 times just to stumble on 'blue' before you get any learning signal at all.”
Dwarkesh Podcast · AI Research & Frontier Labs · May 2026
Context
Teaching out his own 'bits per flop' framework, which decomposes into samples per flop times bits per sample. His point is that long-horizon RL is punished on both terms at once — fewer samples per unit of compute, and less information per sample than supervised learning's cross-entropy signal would give you.
Transcript
Why have they converged to that?
…way and then like just use advantage here. And then the sort of, if there's a problem in these dynamics, it doesn't blow up your loss as much. And so in robotics, there's a kind of convergence towards more like using off-policy data to just shape your rewards, but not actually be directly your.
I'm reminded now of our earlier conversation of why MCTS is so favorable as compared to the kind of reinforce or policy gradient kind of thing LLMs do. And this might be totally wrong, but I wrote a blog post a few months ago about how RL, at least policy gradient RL, is even more inefficient than you might think. And so the inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory in order to get any learning signal at all. And so as these trajectories become longer and longer, as an agent has to, instead of just previously complete the next word in the sentence, it has to go instead to, hey, do two days worth of work to figure out even if you even did this project correctly, the amount of information per flop has been decreasing. As you had to unroll two days worth of thinking in order to see if you even did something correctly, to like, did I implement this feature? The amount of samples per flop has been decreasing. But so you can think of you're trying to maximize as your learning bits per flop, right? And this is, you can think of bits per flop as samples per flop times bits per sample. And what I just mentioned a second ago is that the samples per flop go down as RL becomes more and more along horizon. But at least this kind of naive RL is also terrible from a bits per sample perspective. And here's what I mean, at least compared to supervised learning. So early on in training, let's say you have a vocabulary…
And what's also tough here is that actually the distribution that you're sampling under is your policies distribution. So it's like if your policy has no chance of sampling blue, then you will never get a signal.
Exactly, right, right. So that's being modeled by the fact that your probability of sampling blue is extremely low. If you do sample it, you do learn as much as you would have learned in a supervised learning. In all other cases, like 99.999% of in an untrained model, you're just learning…
Yeah, and arguably you spend all your time here, potentially never even getting a single success, right? Exactly. So it's a sort of depressing plot in the sense that once you're here, it's not at all obvious how you get to here. Once you're here, you have something, but you actually, in many RL…
More from Dwarkesh Podcast
