In the closing segment on the enterprise side of the AI data business, Handshake's CEO says enterprises are far behind the labs in actually deploying AI. He frames it as a big open question for next year.
seem quite far from your normal business.
…trust it. And if the labs core part of their business model is how do our systems actually get used in the real world, you need to have evals for everything. Everything. We get hit up by labs all the time being like, we just are thirsty for evals. We want more and more and more and more and more.
I love the quote of like a new public benchmark is like fresh snow, fresh snow to ski. You know, it's like, I think a lot of the research lately. Yeah, it's a winning brown code. Yeah, it's GNOME. Yeah, it's like, you know, it's, it's come out there and ski some powder. So, but, but I think also we didn't talk on the enterprise side of the house, like a lot of their IP is really evals. Yeah. Like really codifying how processes work internally. I mean, we didn't talk a lot about just like enterprises really seeing high ROI out of AI yet today, but like I'm having hundreds of conversations. I'm on a plane basically nonstop most weeks, meeting with like co-large leaders, large enterprises. And like outside of using like co-pilots in the workplace, like very, very, very few companies outside of SWE itself are actually deploying end-to-end automation with AI. And I think, you know, I see our mission longer term is not only, you know, helping labs continue to change the world in all these positive ways, but it's also helping enterprises like take advantage of this technology. And I'm actually quite shocked with how few companies I can find leveraging Beyond Copilot and building end-to-end AI processes. And that is, I think, going to be like a really big question mark…
Yeah. Would
you guys dip into like held out evals benchmarks of your own? One of the things of hill climbing on benchmarks is you can hill climb on them and they become kind of useless. Some companies like Cognition keep them held out, right? So they'll publish a subset and keep the rest internal. Yeah, it's…
And maybe one more thing to add on feature is that all the tasks we walk through today are single point in time kind of tasks. So for. For example, if we look back on lab essays, a person doing the life sciences task, lab essays is a core atomic unit. Imagine the situation where a person is not…