“If you really dig into the supply chain a little bit, it's some PhD student who made a benchmark as part of their... As the course of them doing their PhD. And so you're kind of outsourcing all of your product thinking to that PhD student who happened to make a viral benchmark.”

…we care about making useful models. So we want models when you put them into people's hands, they have a great experience using. And I think that is the most important metric for any model that we or other people produce. I think that really focusing on the external benchmarks, if you really dig into the supply chain a little bit, it's some PhD student who made a benchmark as part of their... As the course of them doing their PhD. And so you're kind of outsourcing all of your product thinking to that PhD student who happened to make a viral benchmark. John Collison Right. Alexandr Wang So clearly that's wrong. John Collison Yeah. Alexandr Wang And so that's why we care a lot... A huge part of what a lot of people at Meta and MSL are focused on is crafting amazing evals. Mark Zuckerberg And now that we have them, we can work on our own. But if you're building a... You're ramping up your lab, right? It's like those public ones are not a crazy…