Things Adam Says on Podcasts

Yeah, I mean, it's a good question. I don't want to diminish what LLMs are capable of, but I keep thinking about calculators. You know, calculators took something that, and I mean like pocket calculators, right? They took something that like we thought of as this inherent human activity of, you know, compute, like that kind of computation, like for function mathematics, and automated it. And it used to be that to be a good mathematician or a good physical scientist, you had to be good at doing that kind of work with pencil and paper or in your head. And then that stopped being true. And in some ways, you know, that was a loss of certain things. But on the other hand, it allowed for a kind of mathematical research that could not have been done before, right? You know, you'll hear sometimes mathematicians say things like, computers are like telescopes for mathematics. And, you know, but again, I mean, not to repeat myself, but I do think that the difference here is that it's language, not math. And so that makes it feel like something is sort of thinking and conscious and talking to us. And, you know, again, that's not to diminish it. You know, I'm not diminishing the functionality of calculators either. Um, but it is quite deceptive. And I think I, in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day. They are for predicting the next word or whatever in the sequence that they've been given. And it turns out you can get pretty far doing that.

You can. And I would push back a little bit on the stochastic parrot thing, even though that's technically true. I mean, internally, they are acquiring during their training process some kind of coarse-grained abstraction, some kind of structure, which allows them to extrapolate, generalize, call it what you want. So, you know, it's different to us, but there's something there. But I think the $2 million questions are at the moment, it needs human supervision. And these folks say, well, we just scale is all you need. Yeah, at the moment, it needs human supervision. And at the moment, it just generates loads of spaghetti garbage code. And, you know, it just creates, basically it creates more problems than it solves. But it's deceptive because most people can't see the problems. But all we need to do is keep scaling it. You know, when GPT-7 comes out, it will actually refactor all that code. Maybe we can RL train it to refactor it. And we just have to kind of, and it's very dangerous keeping going because now we're messing all of our code bases up and we're creating all of this slop everywhere. But just hold on, boys, just wait for a couple of years and the next version will kind of bring it back in check. What do you think about that?

know. And ironically, people aren't really talking much about the hallucination now because they're agentic and they can fix their own stuff. You know, it's more like, you know, if you analogize it like a database query or a program interpreter, it's only as good as the program or the database query. So, you know, it will basically do what you tell it to do. And this is the gap, right? Because if these things could be alive, if they did have agency, what would that mean? Let's wire it up in a loop and let's show it a load of video frames in a sequence and we'll give it a basic prompt, like do stuff, do something interesting. What will happen? Basically nothing. Nothing interesting will happen, right? So the more you understand a domain and you put a very specific program in there, you can get it to do a specific thing. You can get it to hill climb towards a specific goal to solve a specific problem. But the framing always comes from us. So it's kind of doubtful whether we would overcome that. Maybe we will, but it's kind of doubtful. But then there's this thing, which is the first or second step fallacy, right? Which is, as you said, Eliezer Yudkowski, he said that we're only one or two steps away from inventing AGI and it's going to run away. Why does he think that?

I mean, first of all, he didn't say one or two. He actually said zero to two. He's not sure that we need any. But yeah, I mean, why did he say that? I mean, you should ask him. But he believes in something like a singularity. He believes that if you just throw enough computing power at a machine learning system of the right type, then it will become conscious, wake up, whatever term you want to use, and then become intelligent and then use that intelligence to increase its own power. Which will increase its intelligence further, and this will create a sort of feedback loop, and you'll get an intelligence explosion. And then he believes followed shortly by the end of the world.

Can we unpack this a little bit? So, you were speaking there a little bit about instrumental convergence, which is this idea that basically its instrumental sub-goals towards whatever it's doing will kind of converge on things like power seeking and bad things that we don't want. So, it's always going to kill us all in almost every scenario. Yeah,

Yeah, yeah, that's exactly right. And I do think a lot of it is motivated by fear of death, which, you know, yeah, I don't want to die either, especially not anytime soon. But there's a difference between, you know, a healthy fear of death and letting it sort of run your life. But yeah, I mean, Kurzweil basically takes Moore's Law, right? This exponential increase in the number of transistors that you can cram into the same area on a silicon chip and generalizes it and says, oh, you know, this is not just a technological phenomenon that lasted for about 50 years from the late 2020th into the early 21st century. Instead, this is a general trend in technology, something he calls the law of accelerating returns, and not just technology, but actually biology as well. So he traces it back at least as far as the start of life on Earth and in some places, I think, even claims to trace it all the way back to the Big Bang with the organization of inorganic systems as well. He just thinks that there is a trend toward greater complexity and intelligence that runs through absolutely everything and that it's pointing to a time in the very near future. He has repeatedly said the year 2045 is when the singularity arrives and our technology becomes unimaginably advanced. And the evidence he marshals for this is pretty weak. He says, you know, you can pick out particular points of interest in the history of life on Earth and the history of human technology. And when you plot them on a chart, you get, you know, a straight line on a semi-log chart. And so that means that you've got an exponential trend. But the fact is that, you know, those points are cherry-picked. And he's got a problem that I think most of us have when we look at history. You know, unless you work really hard, history looks sort of logarithmic, right? You've got this clearer view of what's been happening in the recent past. And then the more distant past is more distant. And so you can't see it as well. And so you end up with this sort of, you know, logarithmic view of the past that is something that I think Kurzweil is mistaking for a true exponential trend.

Yes. And I think you gave an example in your book that, you know, you could look at, maybe this was Kurzweil's example. You can look at lily pads and then they grow exponentially and then they cover the size of the pond and then that's it. But more broadly, though, there are limits. Yeah, exactly. Yeah, what interests me is that we've spoken to Chomsky about this and he said after Newton exercised the ghost but left the machine intact, you know, that the whole enterprise of science stopped being about trying to understand how the universe actually worked and it was more about making sense of the universe. I mean, David Krakow said to us that science is. Like poetry, you know, it allows us to give the universe meaning and make it make sense to us. But reality is protein, isn't it? So, what Ray Kurzweil did was he selected a bunch of things that were relevant to humans. So, he didn't select, you know, things that might have been more globally important to our success. And then he kind of fitted it on a graph. And this isn't just this is a very natural thing, right? The world is complicated. And to make it make sense, we kind of select things that privilege the way we think about the world.

But I suppose it is a bit of a straw man just to make the sigmoid case. So what they would say is that the actual exponential curve is the stacking of sigmoids. And in technology, whether it's Betamax or whether it's film processing technology, what tends to happen is you get these disruptions. So you find a divergent stepping stone and usually a different people, you know, different group of people situated somewhere else. They find a new way of doing things and it just keeps going.

Yeah, yeah, yeah, yeah. No, that's exactly what Kurzweil says. He says, you know, the exponential trend is itself made by a bunch of sigmoid curves and each curve lets you get further up the overall exponential trend. Sure, yes, he does say that. But as I say in the book, there is absolutely no reason why that is something that's always going to happen. And indeed, his own sort of generalization of Moore's Law, when he puts that together, it ignores much of the early history of computing where stuff like that did not happen. If you go back to, you know, analog computation devices from 1,000, 2,000 years ago, they don't fit on his trend. And there's no technology coming up that looks like it's going to save Moore's Law in the future. And that's just one trend. Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of, you know, human technology and human growth. And that's, I'm sorry, just wish casting.

Yes, but there does seem to be an obsession with infinity and keeping going. I mean, you gave another wonderful example of Jeff Bezos. And apparently he said he was very, very scared of stasis. So for him, death is the lack of growth, right? Yeah. And there's a wonderful bit in your book, Wait, where you said, you know, okay, so he's saying like we need to keep using more and more energy every single year. And you said, yeah, you know, maybe we can extrapolate this and possibly for another 3,700 years, we can still use more energy. But if you think about it, that's actually less time than when the great pyramid of Giza was created. And at some point, it must end. Yeah,