JL

Jacob Lauritzen

Things Jacob Says on Podcasts

Where to Find Them

Jacob Lauritzen has been a guest on Latent Space (2 times) and The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch .

Recently: “Building AI for Lawyers - Jacob Lauritzen, CTO of Legora” on Latent Space (November 2026); “Building Legal Agents When You Can't See the Data - Jacob Lauritzen, CTO of Legora” on Latent Space (November 2026); “20Product: Inside Legora's Tech Stack: Why Token Maxing is Failing Enterprise Startups with Jacob Lauritzen, CTO @ Legora” on The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch (June 2026).

What They Said

“We had one problem where it wouldn't cite properly and so it would be difficult to follow, and it was like, next model release solved. It went from like 70% to like 99. It's just like, we did nothing, and like, great.” — Jacob Lauritzen, Latent Space

Explaining why the evals team was arguably not worth much until recently: each new model release kept fixing problems the company would otherwise have engineered around. This is his example of a model upgrade fixing a citation problem for free.

Latent Space · 2026-11-02 Permalink → Listen →
Latent Space Around 09:30 into the episode
Jacob Lauritzen

…just got better. So it was like, do we want to spend a lot of our time doing really complex evals or do we just want to build, because there was so much to be gained from building a better harness and just sort of leaning into everything is going to improve and everything kind of just improved.

Shawn swyx Wang

Yeah, like every two, three months, you can just pass it through, right?

Jacob Lauritzen

Exactly. I mean, we had one problem where like it wouldn't cite properly and so it would be difficult to follow and it was like next model release solved. It went from like 70% to like 99. It's just like we did nothing and like great. Yeah.

Shawn swyx Wang

Maybe the real work is keeping all the data private, making it taggable, sortable, adding new features like the tabular view. Is that the you know, you mean

Jacob Lauritzen

over and above ChatGPT?

Shawn swyx Wang

Yes. But obviously now you're also doing evals. I assume your use of open models is also increasing. We can talk about that. So yeah, I don't know what I'm saying here apart from like, you know, what is the job? The job is maybe not in the early days, it was not so much to do evals because it was…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.
“We can't look at any traces basically. We can't look at any data because it's all super confidential. And that's like extremely frustrating to work with because you basically don't know what people are doing. So it's like, here's an agent and it can do so many amazing things and I have no idea what people are actually doing.” — Jacob Lauritzen, Latent Space

Asked how the company knows its product is working, Lauritzen explains that legal data belongs to law firms' clients, so the team cannot read usage traces and has to build eval sets that approximate what users do. This is the core constraint behind the episode's title.

Latent Space · 2026-11-02 Permalink → Listen →
Latent Space Around 22:27 into the episode
Jacob Lauritzen

don't know if that's the reason that they want a custom build versus not custom build. I think they both know that this is like extremely important and key to their firm's success and their corporate success. So it's both extremely important. I think it's just a different way to achieve it.

Basil Chatha

Okay. Another question I think maybe you want to bring this up is like product metrics. Like, how do you know that like, you know, we're actually doing something good here? Yeah. How do you think about that? I'm assuming that's also related to evals.

Jacob Lauritzen

…out generally like do we solve the problem here? Just with like a super cheap LMS address, try to just have an online metric and then we can use that also when we do roll-ups of new things. We can just like compare feature flag wise. Okay, does it did it improve or like did it catastrophically decline? Because one thing actually that's might be interesting that's difficult for our businesses, we can't look at any traces basically. We can't look at any data because it's all super confidential. And that's like extremely frustrating to work with because you basically don't know what people are doing. So it's like, here's an agent and it can do so many amazing things and I have no idea what people are actually doing. I would have assumed

Basil Chatha

you guys like signed some sort of agreement with the law firm and they're like, oh, yeah. I wish.

Jacob Lauritzen

Yeah, I wish. No, but the thing is, like, a lot of the data that they have is not even theirs, right? It's their clients and it's super confidential. So we can't look at it. And so basically, we have an evals team that's like trying to approximate what we think everyone is doing and like build that…

Shawn swyx Wang

Can you do paraphrase? So for example, ChatGPT has like a slash feedback thing that directly fully says the full trajectory. But obviously that is not possible for you guys. But you can use an LM to paraphrase like from this real thing, make a synthetic thing that is like close enough, but change…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

Collections They Appear In

AI Builders Startups & Venture