Things Alexandr Says on Podcasts

…we cared more about what the external benchmarks were just to make sure we were on the right track. And then I'd say, you know, each release that goes by, the larger share of what we're looking at are internal ones. Alexandr Wang And one of the things that we actually quantified in our research blog post as part of Muse Wonder is this metric for big model smell, which is kind of, you know, big model smell and feeling of benchmaxing are kind of like inverses of one another. John Collison This idea that you could just like tell when you're using a really big model, it just... You can tell. Mark Zuckerberg It just has nuance. John Collison It's got that je ne sais quoi. Alexandr Wang Yeah. And so we, the way we quantify that is actually almost explicitly what you're mentioning, which is the gap between in-distribution performance and out-of-distribution performance.…

…probably be able to stay in the lead. So I think that's one factor. I think the power and strength of distillation is certainly another factor that's certainly benefiting the open source ecosystem quite a bit. And I think this is just like a— John Collison There's an anti-hegemonic force in the phenomenon of distilling a model. Alexandr Wang Yeah. And this is just a quality of the technology. It's not like anyone planned for distillation to work super duper well, but it just turns out that you can get a lot of capabilities naturally through some form of distillation. And this is, broadly speaking, distillation is quite a good force for the entire ecosystem because it means that you can pack very strong intelligence capabilities into smaller and smaller models, which enables us to serve them at much wider scale. And that's certainly one of the very exciting stories is just like really small models today are like, you know, much more powerful than the biggest…

…I just want to make it simple. Alexandr Wang Yeah. I think that we've been... I think somewhat infamously, a lot of the AI products have been designed by a highly technical group that do not have these sort of randomly sampled American in their mental model. And so the products... I mean, the naming of some of the models is like so egregious. John Collison The numbers go backwards? Alexandr Wang Some of the times the numbers go backwards and sometimes it's just like a totally different word and you have another set of words. And like, I just like, how is somebody who's not carefully reading API design specs supposed to know what they're supposed to pick? Like, it's just far too confusing. John Collison Okay, we're talking about models a lot. Tell me about the new model. It sounds awesome. Mark Zuckerberg Yeah, I mean, I think we're quite proud of this. Um, yeah, the new model—Muse Wonder—it's a new class of model that is at the frontier. So we've basically been working on this since the beginning. Basically, when you're doing these pre-trains…

…I've been here, but also, you know, as you mentioned, like, long before that, where you have to make very large investments or make very big decisions about, you know, how you're going to invest or develop the technology or, you know, what decisions you're going to make long before there's proof points around whether or not even the thing is working. And so I think this is like, in some sense, it's like entrepreneurship on hard mode. It's like the opposite of Lean Startup. Like, Lean Startup is like, yeah, you like see if you get something working and then you double down and then you see if that works, then you double down. Like, you cannot do that in AI. You have to like weave— I mean, we've been investing into these data centers since, you know, for years and years and years. We've had to— John Collison And you'll be radically behind if you wait to see if things are working. Yeah. Mark Zuckerberg So we have these conversations where it's like, At the end of last year, we're like, all right, we are quite excited about the progress. Yeah, we…

…we care about making useful models. So we want models when you put them into people's hands, they have a great experience using. And I think that is the most important metric for any model that we or other people produce. I think that really focusing on the external benchmarks, if you really dig into the supply chain a little bit, it's some PhD student who made a benchmark as part of their... As the course of them doing their PhD. And so you're kind of outsourcing all of your product thinking to that PhD student who happened to make a viral benchmark. John Collison Right. Alexandr Wang So clearly that's wrong. John Collison Yeah. Alexandr Wang And so that's why we care a lot... A huge part of what a lot of people at Meta and MSL are focused on is crafting amazing evals. Mark Zuckerberg And now that we have them, we can work on our own. But if you're building a... You're ramping up your lab, right? It's like those public ones are not a crazy…