“If we're in 2036, it's been 10 years, and we don't have billions of crazy superintelligences running around that have radically transformed the world — other than exogenous political shocks, a war, or a ban on AI — what is the most likely technical reason that doesn't end up being the case?”

I mean, like, my reason would just be like, it's got to be this sort of like there's been a classic thing almost like Marvex Paradox, right? Where like we see like, you know, we think of the AI being like, if it can do this, it's going to be amazing, right? Like if it can solve these hard mouth problems, if it can win a chess, blah, blah. And then it solves these things and then it's like not that impactful. Obviously, it's somewhat impactful, but like not everything. It's like if somehow that continues and like there's never like the true like spark of generalization that occurs, I think that could lead to like the AI just being like extremely good at kind of everything that people like put into a benchmark, put into an environment, but like there's still some persistent like sim to wheel which is somehow blocking everything. I think this is kind of unlikely. I think we do actually see this kind of generalization even from RL in practice already. But if it is just ridiculously hard to generalize meta-learning, plus we don't solve continued learning and it's just super hard and impossible. This would be my default scenario in that case.

Yeah, I agree with that. Humans have a lot of advantages over models now. And each time a new model comes out, it'll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment or the models can't check themselves well enough. Yeah, so there's this cycle that keeps repeating where people think where a new model comes out and people are blown away and they're like, this is it. This is AGI. But then they use it a bit and then it starts to feel dumb after a month or so. So that cycle just might keep going and it's hard to predict how many times it's going to repeat. And right now, you don't get explosive growth in capabilities because you still get bottlenecked enough when you're trying to do research in engineering that even if the model can write way more code than a person, it doesn't make you 100 times more productive. But yeah, so maybe there are just more of these cycles than we would expect.

For me, it's like a question of how far off this global optimum of a learner you could have on a chip is like the transformer plus like RL, basically the current recipe. So I think people imagine that even once you have an agent which is better than all humans at AI research, even if it's 0.1% better than all humans, then the fact that you can run hundreds of thousands, if not millions of these in parallel, you can run them much faster, chip's going to speed up, that's going to outweigh every other bottleneck. And you're eventually just going to hit this very fast takeoff with recursive self-improvement. I could imagine that if we continue along the trajectory that we're currently on with that paradigm where it's basically just self-attention, RL, scaling up RL environments, I guess if you think about what happened with Moore's Law, we had this very nice straight line and that held for a really, really long time. But there were so many discrete discontinuities and innovations that had to happen to keep that scaling law going. And the same thing has kind of happened with LLMs. We had this pre-training scaling law, and then that was kind of like hitting the diminishing returns. And then we came up with RL and solve that. And then we got this new diminishing returns curve to hit that made it keep looking like a straight line going up. And so if it requires another one of those discontinuities to solve, I'm not sure that the current method of training LLMs with these RL environments, even RSI targeted RL environments, would be able to discover that discontinuity. And if not, we're probably going to hit this asymptotic curve where...