Derek Thompson asked why so many people influenced by effective altruism warn about AI danger while building frontier AI. MacAskill says the labs made more sense as research outfits; his worry is that commercial competition will stop them acting once their systems become truly dangerous. Thompson replied that the analogy is too generous.
…And that's really a rate of progress that I think the world has a rate of tech development that the world has never seen before. And my guess is that if the accelerationists thought that that was what was going to happen, they'd think that maybe we ought to be a little bit cautious about it too.
…Please slow us down. Let's slow. Let's pause. Let's pace the frontier. And then on Tuesday, we just made a giant leap forward in scientific and mathematical progress. I mean, there is something unbelievably incongruous and potentially insane that's happening here. And I wonder what you make of it.
Yeah, I mean, I think it's just very confusing for the general public. And I honestly even sympathize with people thinking, well, surely they can't possibly believe this because they wouldn't be building it. I mean, the analogy is like, imagine in the 1950s, you'd realized that fossil fuels produce climate change via carbon dioxide emissions. And so you thought, okay, well, the thing we've got to do then is build oil companies, build coal companies so that we can steer this in a positive direction and accelerate the clean energy transition. And, you know, maybe that argument works, but it's certainly a weird, like it's certainly a weird move. Now, the situation is a little different in the case of AI because we have not yet crossed the Rubicon where the AI systems actually pose meaningful existential risks. So in the early days, OpenAI got started as a non-profit research lab, and Anthropic got started off as a spin-off that was basically, again, going to be a research lab. And then that makes sense. You want to understand the risks from AI and you want to be able to make sure we align AI systems. Well, you've got to be working on it in order to be able to do that.…
Yeah, I mean, I understand where you're going with your Exxon analogy, but I think even that doesn't take into consideration the thing that you just said makes AI most dangerous, which is recursive self-improvement, AI that builds and Controls itself and escapes human monitoring and oversight. What…
Yeah, my number one favored policy that the companies could implement is about what fraction of their computing power they devote to research and development and future training, and what fraction they develop to devote to infants that is serving customers or to work on safety and alignment, where…
You mentioned that effective altruism is not worried about slightly crazy ideas. So let's end with a slightly crazy idea. One of the things I think we're going to hear more about in the next few years as these models get more powerful is the question of AI consciousness, as insane as those…