Asked how practitioners should evaluate AI agents, Shankar says exploratory data analysis doesn't carry over to unstructured agent outputs yet. For now, reading outputs one at a time is all anyone has.
Yes.
This is, you know, to a large extent, like the data engineer, analytics engineer toolkit is just as relevant pre-agents as post-agents. But this is a brand new thing. Do you have recommendations for practitioners on practically how to go about doing this well?
…my outlines? You can define this in your favorite database framework, data framework, but you cannot define this on your AI agent outputs. So we don't have the tools yet to accelerate all of these patterns that we know of how to work with data. And I think that's where people get stuck because now that's like they do have to invent from first principles of like, how do I look at this data? Like imagine if you had to do EDA and you had no pandas functions or you had none of these like df.summary or df.visualize. What would you do? You would literally load up the table in Excel and you would look row by row and you would be like, I don't know what I'm doing. Right. But that's the analog of how we're working today, right? We look at a spreadsheet of LLM outputs and we look row by row and we hope and pray for the best. We try to define some rubric and we're like, okay, maybe I'll run this rubric at scale on the outputs with another LLM judge and then look at that results in a spreadsheet. And like…
I love that. It's an indication of just where we are in figuring out the new rules of the road, which is fun for me. I like transition points because it means that there's stuff to learn. And that's when it gets interesting. Speaking of which, last question for you. Wait, you're headed to or you're…
I am headed to. So when do you start? 2027.
We talked a lot about the research that you've done. What's top of mind? What are you going to be, what are you going to get up to?
