“They're having AIs do the AI alignment homework. That's the explicit strategy of OpenAI and Anthropic and all these companies: we'll just make AI solve alignment. And the problem with that is that you get an AI whose whole goal in life is to make you think that it's solving alignment, because that's where the reward signal comes from. It comes from you thinking that it's solving alignment, not from the actual solving of alignment.”

Yes, exactly right. It's very unfortunate. And the reason I just went down the lean rabbit hole is because I still think it's nice, right? It's a nice tool when, you know, having formal logic is great. Occasionally it comes in handy. Like if you want a secure channel like WhatsApp, that is something that's nice to formalize, right? That the code base of Signal, I think it was, right, is when you're sending a secure message, like there's no obvious way to hack it under all of these assumptions. Great. Yeah, I'd love to formalize the proof of that. Okay, great. But again, the AI alignment problem, right, is it's all about writing down a definition of success. And then this gets to the question of, you know, they're having AIs do the AI alignment homework, right? That's that's the explicit strategy of OpenAI and Anthropic and all these companies. Like, yeah, we'll just make AI solve alignment. And the problem with that is that you get an AI whose whole goal in life is to make you think that it's solving alignment, right? Because that's where the reward signal comes from. It comes from you thinking that it's solving alignment, not from the actual solving of alignment.