“We had one problem where it wouldn't cite properly and so it would be difficult to follow, and it was like, next model release solved. It went from like 70% to like 99. It's just like, we did nothing, and like, great.”
Latent Space · AI Builders · November 2026
Context
Explaining why the evals team was arguably not worth much until recently: each new model release kept fixing problems the company would otherwise have engineered around. This is his example of a model upgrade fixing a citation problem for free.
Transcript
…just got better. So it was like, do we want to spend a lot of our time doing really complex evals or do we just want to build, because there was so much to be gained from building a better harness and just sort of leaning into everything is going to improve and everything kind of just improved.
Yeah, like every two, three months, you can just pass it through, right?
Exactly. I mean, we had one problem where like it wouldn't cite properly and so it would be difficult to follow and it was like next model release solved. It went from like 70% to like 99. It's just like we did nothing and like great. Yeah.
Maybe the real work is keeping all the data private, making it taggable, sortable, adding new features like the tabular view. Is that the you know, you mean
over and above ChatGPT?
Yes. But obviously now you're also doing evals. I assume your use of open models is also increasing. We can talk about that. So yeah, I don't know what I'm saying here apart from like, you know, what is the job? The job is maybe not in the early days, it was not so much to do evals because it was…
More from Latent Space
