“We can't look at any traces basically. We can't look at any data because it's all super confidential. And that's like extremely frustrating to work with because you basically don't know what people are doing. So it's like, here's an agent and it can do so many amazing things and I have no idea what people are actually doing.”
Latent Space · AI Builders · November 2026
Context
Asked how the company knows its product is working, Lauritzen explains that legal data belongs to law firms' clients, so the team cannot read usage traces and has to build eval sets that approximate what users do. This is the core constraint behind the episode's title.
Transcript
don't know if that's the reason that they want a custom build versus not custom build. I think they both know that this is like extremely important and key to their firm's success and their corporate success. So it's both extremely important. I think it's just a different way to achieve it.
Okay. Another question I think maybe you want to bring this up is like product metrics. Like, how do you know that like, you know, we're actually doing something good here? Yeah. How do you think about that? I'm assuming that's also related to evals.
…out generally like do we solve the problem here? Just with like a super cheap LMS address, try to just have an online metric and then we can use that also when we do roll-ups of new things. We can just like compare feature flag wise. Okay, does it did it improve or like did it catastrophically decline? Because one thing actually that's might be interesting that's difficult for our businesses, we can't look at any traces basically. We can't look at any data because it's all super confidential. And that's like extremely frustrating to work with because you basically don't know what people are doing. So it's like, here's an agent and it can do so many amazing things and I have no idea what people are actually doing. I would have assumed
you guys like signed some sort of agreement with the law firm and they're like, oh, yeah. I wish.
Yeah, I wish. No, but the thing is, like, a lot of the data that they have is not even theirs, right? It's their clients and it's super confidential. So we can't look at it. And so basically, we have an evals team that's like trying to approximate what we think everyone is doing and like build that…
Can you do paraphrase? So for example, ChatGPT has like a slash feedback thing that directly fully says the full trajectory. But obviously that is not possible for you guys. But you can use an LM to paraphrase like from this real thing, make a synthetic thing that is like close enough, but change…
More from Latent Space
