Almeida argues that the real test of AI is whether it can automate ordinary productive work, not whether it can top hard benchmarks. Casado pushes back right after, suggesting the real world is long-tailed and the training data for it simply isn't there yet.
And so you think that the measure that we should have is to what extent can you automate actual productive tasks? That the overpromise and under deliver, that's
the dimension in particular you're talking to. The ability to automate tasks.
I think in my heart, it's like cool sci-fi. And I think that that is the canary in the coal mine for cool sci-fi. Like, are you really telling me that math is solved or like even two years ago, GPQA, that Google proof question answering is solved, but we still can't handle a drive-through, right? Like it's a very hard thing to hold in your head at once. And I think a lot of people don't have good answers to that.
Can I just test one thing, which may not make sense, but I want to say, I mean, isn't there an argument, though, that like the distribution of the real world is different than the digital world, right? It's heavy-tailed. There's a lot of exceptions. We don't have all the data. And I mean, couldn't it be the case that the reason we're not doing productive stuff in the real world is just like we're not, we don't have the data for that distribution. We're not training on that distribution. And this is why it's just been basically relegated to like these lower dimensional manifolds, like whatever math or code or
I don't entirely buy the data argument, in my opinion. I do believe that there's a long tail for sure. Like that, that would be kind of crazy to deny. And I don't think that in my like canary in the coal mine situation, we need to automate that long tail. Like I think that I think we need to be incredibly pragmatic on everything. And like building reliable software is always an investment, right? Like it like, you know, what were the three great virtues of a programmer? Laziness to not to do it again, hubris. And there's a third wall.
Yeah, no, I remember this is from the Pearl days. Yeah, yeah.