JH Jeremie Harris Jeremie Harris Physicist-turned AI startup founder who co-founded the AI safety company Gladstone AI and the AI mentorship marketplace SharpestMinds.

“Evals are fake. Evals are fake. If you take one thing home, anybody who shows you an eval to make you feel calm and happy about the safety of their model is full of shit. There is no such thing as an eval that can actually tell you that a model is safe.”

Last Week in AI · AI Research & Frontier Labs · November 2026

“Evals are fake. Evals are fake. If you take one thing home, anybody who shows you an eval to make you feel calm and happy about the safety of their model is full of shit. There is no such thing as an eval that can actually tell you that a model is safe.” — Jeremie Harris, Last Week in AI

Context

The hosts were discussing a reported draft AI safety bill under which the government could block a model only if it posed a foreseeable, imminent risk. Harris argues nobody can show that, because safety evals can't reveal it.

Transcript

Last Week in AI Around 59:16 into the episode
Jeremie Harris

…they've taken, the incentives that apply to them. And the numbers themselves, I think, are the most important and interesting part of this. We're still in a phase where it's a mixed story. To your point, I almost kind of want to say I'm leaning towards, you know, hey, DAUs are still climbing.

Andrey Kurenkov

…going to say I'm excited or very optimistic around something like this passing at the federal level. But it is interesting that Afropublic has a more specific kind of bill that they are pushing for. I think previous year have been a bit more kind of big picture in terms of their recommendations.

Jeremie Harris

…poses an imminent risk like this. That's almost like the foreseeability of this, depending on the exact legal instantiation, it's foreseeable. I hereby foresee it for some level of capability that we're approaching pretty damn soon. I can't tell you specifically which one, how many oons, but it's coming for us. But I have to be able to apply it to a specific model here, and no one knows. Like evals are fake. Evals are fake. If you take one thing home, anybody who shows you an eval to make you feel calm and happy about the safety of their model is full of shit. There is no such thing as an eval that can actually tell you that a model is safe.

Andrey Kurenkov

Evals are requirement. I forget, it's like sufficient, but not complete, right? Like necessary, but

Jeremie Harris

not sufficient. Necessary but

Andrey Kurenkov

not sufficient. So that tells you if you're doing bad on the evals, you know, you know it's bad. Yeah. If you're doing good on the benchmarks, you don't know, right? It might be okay, but it might not be okay. And I think we've discussed before, like, there's kind of probably no way to eval to know…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Last Week in AI