“Big model smell and feeling of benchmaxing are kind of like inverses of one another.”

…we cared more about what the external benchmarks were just to make sure we were on the right track. And then I'd say, you know, each release that goes by, the larger share of what we're looking at are internal ones. Alexandr Wang And one of the things that we actually quantified in our research blog post as part of Muse Wonder is this metric for big model smell, which is kind of, you know, big model smell and feeling of benchmaxing are kind of like inverses of one another. John Collison This idea that you could just like tell when you're using a really big model, it just... You can tell. Mark Zuckerberg It just has nuance. John Collison It's got that je ne sais quoi. Alexandr Wang Yeah. And so we, the way we quantify that is actually almost explicitly what you're mentioning, which is the gap between in-distribution performance and out-of-distribution performance.…