“All large language models, the benchmarks that they get benchmarked against, if you get the answer right, you get a point. If you don't get the answer right, you don't get a point. But also, if you don't know the answer, you don't get a point. So the models are incentivized to guess. If you asked a model, what's my date of birth? Mathematically, there's a one in 365 chance that it's going to guess it right. If it says, I don't know, it's got 0% chance of getting a point. And so it's incentivized to make one up, because mathematically, I'd rather take that chance and have a 0.000 whatever percent likelihood of me getting this right versus not getting anything at all. I think that we need to rethink benchmarking in totality in regards to how we evaluate these models.”
sputnikintelligence.com
The panel is discussing a paper on cultural riddles as a way to test AI models, and host David Zax asks Ash about benchmarks. Ash widens the point beyond riddles: he argues that standard benchmark scoring pays a model for guessing and gives nothing for admitting it doesn't know.
…is riddles are, especially for models, can be incredibly difficult to solve since it's not just about pattern matching. You need creative thinking and wordplay to solve most riddles. And so when we're testing the effectiveness, efficiency of a model, riddles are a really good benchmark for them.
…able to dig into the paper at all? And what do you think happens when you both kind of brain tease a model with respect to these riddle-like stretching of language and then also kind of demand of it cultural specificity? What do we learn about models here? I know you love benchmarks, by the way.
I do indeed. David, so first off, you can actually go to Cohere's Cultural Riddles paper website and you can do a quiz. So you can kind of do a quiz and test your knowledge against an AI. Thankfully for me, I outbeat the AI. I think I got a score of like 80% or something. It was actually pretty fun. I recommend you can go and let anyone wants to just go to their website and do it. I hope that they're collecting that as training data. What this exposes is far bigger than a riddle or, you know, like cultural awareness. What this exposes is something that even OpenAI probably put some research out about, I think, last year or the year before. It's that we just like the entire space of benchmarking needs to evolve, right? And let me explain why I say that, right? All large language models, the benchmarks that they get benchmarked against, if you get the answer right, you get a point. If you don't get the answer right, you don't get a point, right? But also, if you don't know the answer, you don't get a point. Okay. So the models are incentivized to guess, right? As opposed to, because they think about it and they go, you know, if you asked a model, what's my date of birth? Okay. Mathematically, there's like a like, there's like a, if you just think about, you know, the day of the year, okay, there's like a one in 365 chance that it's going to guess it right. If it says, I don't know, it's got 0% chance of like getting a point, right? And so it's incentivized to like, oh, I'll just make one up because mathematically, I'd rather take that chance and have…
Last question, Jessica, I'll keep it open-ended. What interested you most in the cultural riddles study?
Oh, what interested me most, I guess it was, I was thinking back to a conference that actually ran this week from IBM Tech Exchange. I'm not sure if you caught this. So Tim O'Reilly is one of the speakers and he talked, actually, he commented on, he made a parallel between words and essentially the…
I love the Brighton spells. I think the ultimate benchmark. Should be expecto-patronum.