“Imagine you have a customer satisfaction score, and you tell your superintelligence, just make that score better. The AI thinks about the fastest, cheapest way and says, okay, I'll create 50,000 bots that call our service hotline and give a five out of five. The score went up. So you say, no, I meant real customers. And the AI thinks, easy, I'll give a $1,000 gift certificate to everyone. And you say, that's not how I meant it. And it's like, but that's what you said.”

That's exactly right. I don't think you want to do just like this crazy breath-first search. You know, I think that will be too slow. Like biological evolution took billions of years. But technological evolution is much, much faster. And there is direction. There's the settings of rewards and environments and sandboxes and so on that you give to an AI and you guide it into the right direction. And what we're seeing is like, you know, as these systems get better and better, they need less and less specific guidance and more and more just very well-defined reward. And so rewards. And so I think actually we'll see this job of a reward engineer become more and more important. And in some cases, the reward engineer can have like pretty benign sort of reward hacks. In other cases, the reward hacks are pretty bad. Like if you work on ExploitBench, the way OpenAI did when the Hugging Face thing happened, then you probably have worse reward hacks than if you work on a kernel optimization algorithm. And where we also ended up finding certain bugs in some production BUDA kernels that we eventually published. And as part of that system, there are a few bugs that were like in very rare circumstances would create some numerical instabilities. And the algorithm kind…