“Hydro is developed now mostly via agents, humans driving agents, and we are putting up PRs at a much faster pace, and the PRs themselves are much larger. Last week we developed a fundamentally new approach to exploring retries that happen in a distributed system. This is like a 5,000-line PR because it's such a large component and all the pieces fit together. It's very hard to break down. So how do we ship these features with confidence that they will compose correctly with other features in the system?”
sputnikintelligence.com
Asked to back up his keynote claim that code review can't keep up with agent-written code, Shadaj gives a concrete case from his own team at AWS. It explains why his project tries to catch mistakes in the compiler instead.
Yeah, that's a big thing. They pay off
the student owns. And it is good. So let's talk about your keynote. So you're giving a keynote here, right? In your keynote, you say that code review can't keep up with how agents write code. Can you put that, I hear that a lot. Can you put that into context for me? What are you seeing?
Yeah, I mean, so Hydro is developed now mostly via agents, you know, humans driving agents. And agents have amplified our ambition in what we can build into Hydro. Now, that has also meant that we are putting up PRs at a much faster pace, and the PRs themselves are much larger, right? Like last week, we developed a fundamentally new approach to being able to explore retries that happen in a distributed system and reason about correctness with us. This is like a 5,000 line PR because it's such a large component and all the pieces kind of fit together. It's very hard to break down. And it's that type of thing that, you know, you might have, it might have been too large in scope to even consider before agents. And so I think one of the big challenges for us is a system like Hydro, we are giving it to developers. Developers trust us that Hydra is doing the right things, that when we say Hydro is checking this correctness property, it will do that. And so, you know, our burden is pretty intense to make sure that the code is correct. And yet we want to accelerate our development. So there's this real challenging trade-off of, you know, how do we ship these features with confidence that they're going to work correctly? They will compose correctly with other features in the system, right? You…
Yeah, well, give me some practical advice. If I was thinking about starting, let's say, an AI software factory, which is the newest thing, right? And so you're suddenly relying on agents to do a whole lot more than normal. What advice would you give to somebody like that?
Yeah, I think a lot of our efforts, and this is what I talk about in the keynote, is finding ways to put guardrails around the agent that are outside of the kind of zone of trust of an agent. So the way I talk about it is, normally your agent is responsible for writing your code and also the unit…
Right. Is there a repository or is there a source that you look at when developing these guardrails?