“When I learn how an LLM works and how simple RLVR is as an algorithm, I'm sort of stunned by the kinds of things it can do — that it can learn to build very complicated code repositories simply from getting a yes or no. And here, I feel like AlphaGo actually seems less impressive in retrospect the more you understand it — because you're putting in a lot of bias, you're telling it how to titrate exploration as things go on, you're building this very explicit tree search for it.”

I'd be curious. How much compute you save by training the value and policy on the same network that because they share the same representations, how much more efficient learning is? Because that would be interesting if they're basically kind of we've just talked about how they're kind of making similar predictions or they should be in line with each other. And so I'd be curious if like actually, yeah, you just halving the amount of compute you got to do by keeping in the same network. Right.

AlphaGo Lee, the original AlphaGo paper, had two separate networks. And then in all subsequent papers, they merged them into two heads. And presumably this saves compute. But answering that question in a very rigorous scientific way is actually, it's a simple question, but in practice, actually takes, like, if you really want to chase that question down to its limit, it takes quite a bit of work to, you know, really resolve that. But intuitively, yes, they share a lot of representations. So, and as we mentioned, there is a sort of like your policy network and your value network when doing evaluation should kind of agree, right? So there really should be this sort of consistency between them. Yeah.

I personally disagree. I think they're profound for different reasons. And I don't understand the LMRL like enough to kind of comment on your podcast about it. But I think AlphaGo, so yeah, why is it a profound accomplishment? I think maybe it's worth stepping back a little bit and just like, it is different than modern RL. And we can talk a little bit about like some of the algorithmic choices there. But I think the most profound thing here is that a 10-layer neural network pass, so basically 10 steps of reasoning. And of course, the reasoning is not just one trail of thought. It could be like the distributed representations and a lot of thoughts going on in the same time. But by construction, let's say a 10-layer neural network can only do 10 sequential steps of thinking. 10 steps of neural network parallelized distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem. So this was a breakthrough that I think most people don't even understand today, like fully comprehend how profound that accomplishment is. And this is what also GERDs like AlphaFold, for example, right? Like where you have a very, very difficult physical simulation process that you would need to roll out so many micro-scale simulations. And yet 10 steps of a somewhat small neural network can somehow capture what feels like a MP class problem into a single problem. And so it actually makes me wonder if our understanding of problems like P equals NP or these very fundamental computational hardness problems are incomplete. It's not like, obviously, this is not a proof of P equals NP or anything, but there's something to it that is very disturbing where what felt like a very hard problem can fall to a very, very simple macroscopic simulation.

That is a very interesting insight that a lot of problems which are proven to be NP hard, like I don't know if Go is proven to be NP hard, but protein folding, et cetera, have been like neural networks can solve them because they're NP hard in the worst case, but we're not dealing with the worst. We're usually not concerned with the worst case. These problems have a lot of structure to them. Yeah,

I think that the kind of question we should be asking ourselves is like, we've been formulating solutions to NP hard problems as a kind of worst case complexity. And I wouldn't say this solves Go, right? It doesn't give us an exact solution of the optimum. But in practice, it is extremely useful. And the same thing has been shown in Alpha Tensor, AlphaFold, where, yes, there is a very hard problem that in the worst case seems intractable, and yet we're able to make almost arbitrary amounts of progress. So here's a sort of like, you know, in the limit, what might this look like, right? Well, if you want to simulate something very complex like weather or predict the future, like, you know, do we live in a simulation or not? The computing resources you need to build a very Complex simulation might be much smaller than you think based on our ability to amortize a lot of that computation into the forward pass of a single network. Interesting. So to me, yeah, AlphaGo was the first paper that kind of like really showed this like profound level of simulation being compressed into a small amount of