Asked for a concrete example of what coding agents do for production code, Naous describes a feature he recently shipped himself: "auto-promotion" in a data lake, which turns every field in semi-structured log events into its own queryable column.
So the latest project that I shipped was,
by the way, this is like, to be clear, this is production code. Yeah, this is production code. So the statement you're about to make is the gains that you get for like shipping production.
So I'll give you the example of the last, there's the last PR that I worked on, or like the last set of PRs, was what we call auto-promotion in a data lake. So in a data lake, you store all the data. And typically the data is column-oriented, parquet files, and iceberg, and so on. And when you get data from logs, they're semi-structured. They're arbitrary columns and so on. In order to have the best layout of the data and to be able to query it efficiently and fast, you kind of, the ideal situation is that every field in the log event becomes a column that you can filter on independently of the rest of the rows. And so we call that auto-promotion or shredding. And doing something like that is incredibly complicated, right? Like you have to go and figure out how does the metadata system work? How do the commits of the metadata system work? How do you coordinate across different writers and so on? And that took maybe about 10 days. In previous days, that would have taken maybe about six months for a single person.
Wow. And was this just you?
That was just me. It is insane.
Yeah. So like factor of 10, factor of 20 improvement.