Yeah, I mean, it's a good question. I don't want to diminish what LLMs are capable of, but I keep thinking about calculators. You know, calculators took something that, and I mean like pocket calculators, right? They took something that like we thought of as this inherent human activity of, you know, compute, like that kind of computation, like for function mathematics, and automated it. And it used to be that to be a good mathematician or a good physical scientist, you had to be good at doing that kind of work with pencil and paper or in your head. And then that stopped being true. And in some ways, you know, that was a loss of certain things. But on the other hand, it allowed for a kind of mathematical research that could not have been done before, right? You know, you'll hear sometimes mathematicians say things like, computers are like telescopes for mathematics. And, you know, but again, I mean, not to repeat myself, but I do think that the difference here is that it's language, not math. And so that makes it feel like something is sort of thinking and conscious and talking to us. And, you know, again, that's not to diminish it. You know, I'm not diminishing the functionality of calculators either. Um, but it is quite deceptive. And I think I, in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day. They are for predicting the next word or whatever in the sequence that they've been given. And it turns out you can get pretty far doing that.

You can. And I would push back a little bit on the stochastic parrot thing, even though that's technically true. I mean, internally, they are acquiring during their training process some kind of coarse-grained abstraction, some kind of structure, which allows them to extrapolate, generalize, call it what you want. So, you know, it's different to us, but there's something there. But I think the $2 million questions are at the moment, it needs human supervision. And these folks say, well, we just scale is all you need. Yeah, at the moment, it needs human supervision. And at the moment, it just generates loads of spaghetti garbage code. And, you know, it just creates, basically it creates more problems than it solves. But it's deceptive because most people can't see the problems. But all we need to do is keep scaling it. You know, when GPT-7 comes out, it will actually refactor all that code. Maybe we can RL train it to refactor it. And we just have to kind of, and it's very dangerous keeping going because now we're messing all of our code bases up and we're creating all of this slop everywhere. But just hold on, boys, just wait for a couple of years and the next version will kind of bring it back in check. What do you think about that?

know. And ironically, people aren't really talking much about the hallucination now because they're agentic and they can fix their own stuff. You know, it's more like, you know, if you analogize it like a database query or a program interpreter, it's only as good as the program or the database query. So, you know, it will basically do what you tell it to do. And this is the gap, right? Because if these things could be alive, if they did have agency, what would that mean? Let's wire it up in a loop and let's show it a load of video frames in a sequence and we'll give it a basic prompt, like do stuff, do something interesting. What will happen? Basically nothing. Nothing interesting will happen, right? So the more you understand a domain and you put a very specific program in there, you can get it to do a specific thing. You can get it to hill climb towards a specific goal to solve a specific problem. But the framing always comes from us. So it's kind of doubtful whether we would overcome that. Maybe we will, but it's kind of doubtful. But then there's this thing, which is the first or second step fallacy, right? Which is, as you said, Eliezer Yudkowski, he said that we're only one or two steps away from inventing AGI and it's going to run away. Why does he think that?

I mean, first of all, he didn't say one or two. He actually said zero to two. He's not sure that we need any. But yeah, I mean, why did he say that? I mean, you should ask him. But he believes in something like a singularity. He believes that if you just throw enough computing power at a machine learning system of the right type, then it will become conscious, wake up, whatever term you want to use, and then become intelligent and then use that intelligence to increase its own power. Which will increase its intelligence further, and this will create a sort of feedback loop, and you'll get an intelligence explosion. And then he believes followed shortly by the end of the world.

Can we unpack this a little bit? So, you were speaking there a little bit about instrumental convergence, which is this idea that basically its instrumental sub-goals towards whatever it's doing will kind of converge on things like power seeking and bad things that we don't want. So, it's always going to kill us all in almost every scenario. Yeah,