“I think that if we re-architect our software so that we can understand it at the level of functions instead of lines, if you re-architect it so that there are bounded parts that can be cut. And of course, the most durable part of this system would be the evals, not the code itself. The evals describe what must be true for this to be defined as good. And then, yeah, we can generate 10,000 … and see which one is the best, you know?”
sputnikintelligence.com
Charity Majors has just argued that AI can now produce code at roughly median-engineer quality, so code can be generated in huge numbers and compared. Here she describes how software would need to be re-architected so people can still understand it. She says the evals, which define what good looks like, would last longer than the code itself.
…at the same level of quality as the median software engineer, which means that while one software engineer can output a piece of work that works, AI Can generate a thousand of them, 10,000. We can see which one is the best. You know, it just, it makes you think very differently about code itself.
It's interesting where it's where you spin up all these nodes, now you can spin up all these versions of that feature and compete them against each other.
That's one of the building blocks, I think, because it's pretty clear to me that in the same way, we don't expect every software engineer to understand how to debug types of RAM or how many CPUs or if there's a device. We just don't expect them to understand that. We used to. We don't anymore. And I don't think it's an exact analogy, but I do think that when we talk about cost of software as defined by its maintenance, we're talking about the cost of having to understand everything. And I think that if we re-architect our software so that we can understand it at the level of functions instead of lines, if you re-architect it so that there are bounded parts that can be cut. And of course, the most durable part of this system would be the evals, not the code itself. The evals describe what must be true for this to be defined as good. And then, yeah, we can generate 10,000 disks and see which one is the best, you know?
Yeah, yeah. And for those evals, are you thinking like the LLM as a judge sort of evals, or do you have a deterministic eval?
Well, both. And I mean, a lot of these tools actually already exist. Conformance testing. They're all in the realm of ops and QA, which are historically like the cost centers. You're not quite as cool as software engineers. But like they're aimed at describing the world as it is instead of the…
Jez Humble had a talk yesterday, sort of specifically targeting the Strangler Fig architecture.