Jonathan Stark applies the Gell-Mann amnesia effect to AI: you only notice a model's mistakes in the areas where you are the expert.
…it recently where when I was a little kid, I went to a rock show, like a paleontology kind of rock show. And because I was a nerd even then, and a reporter was there with a photographer, and they interviewed me, took my picture, put it in the paper the next day, and it had like six major errors.
Yeah.
And I was like, I didn't get the, I think, I think it's Gelman amnesia. I don't remember the exact name. I didn't get Gelman amnesia. I never trusted the newspaper again. But most people are like, oh, this one article in the newspaper I'm reading that I know a lot about is totally wrong. It's completely, or at least it's 101 level, simplistic stupidness. But all these other articles are expert level genius articles that don't have typos or don't have the names wrong or don't have a correction in tomorrow's edition, which I'll never see anyway. And I mean, when you're on a daily deadline, and I'm from a newspaper family, a lot of stuff's wrong. But anyway, that's the, that's the, I think that was the original use case, but it's the exact same thing or the original impetus for that study was newspapers. But now it's, it is totally true with Claude and or any of them, any of even the most advanced ones. It is so bad at stuff that I'm good at. Yeah. So why is it, why do I think it's so good at stuff…
Yeah.
Anyway, so I'm just sort of, I'm sort of singing your song here. So what's the, what are the common failure modes for folks you're not just in general, but folks you actually work with, like folks you talk to, what are the failure modes for people who are, you know, pretty serious and maybe have…
Yeah. So by far the most, the biggest mistake I see or the most frequent mistake I see now is people using Claude to create their own backtesting software. So there's lots of backtesting software out there that works. And a much better approach is have teach Claude how to use the existing…