“This is why I believe the superintelligence existential risk discussion is not just distracting, but dangerous. Because with automation bias, with over-trusting these machines already, if you believe that they are nearly superintelligent instead of error-prone large language models parroting things, then you're going to be less likely to check. You're going to be less likely to put checks and balances and regulation.”

To parrot is to repeat back without understanding, right? So there was this whole, this whole existential risk narrative was happening back then too, if you can believe it. And so there was all this conversation about how OpenAI had claimed that GPT-2, the precursor to GPT-3, that powers chat GPT was too dangerous and too powerful to release. There was this conversation about whether GPTs can be ethical or they're creative and all this stuff. And so we really wanted to ground the conversation in the real issues. One of them, one of these issues is the environmental catastrophe, which a lot of people are now seeing, but we discussed it back then. And that was one of the main sections that Google people were unhappy with, the environmental cost. The other one is not documenting your data because you say you have too much data to document. The other one is deceiving people into believing that there is a mind behind the textual outputs that they're interacting with. And so there, that's where we really wanted to explain that these systems are parroting the patterns of their training data. And it's very dangerous when you're outputting text like that because when you have a plausible sounding test or very fluent text, there's so many different kinds of issues that can occur besides you believing that there's a mind behind a machine. I gave in that paper, we gave an example of this Palestinian man writing good morning, which was translated to attack them. And because of that grammatical correctness and there were no cues that the translation could be wrong and people believed the translation. So the other issue of believing there is a mind behind the machine is what we call automation bias. You overtrust automated systems. And if you believe that this thing is an all-knowing machine, then you're going to overtrust the errors that you get, right? So we're seeing this with medical scribes where I was just reading an article, another article where medical scribes that were summarized said this woman was micro-dosing ill mushrooms. School woman has never heard, never discussed mushrooms, never done mushrooms. She saw it on the notes. So none of the doctors, nobody, nobody checked. Because again, if you believe, this is why I believe the super intelligence existential risk discussion is not just distracting, but dangerous. Because with automation bias, with over-trusting these machines already, if you believe that they are nearly super intelligent instead of error-prone large angel models parroting things, then you're going to be less likely to check. You're going to be less likely to put checks and balances and regulation. And you're seeing things like misdiagnosis based on medical errors and things like that, which are serious.