The New Stack Podcast · Hands-On Engineering Podcasts · November 2026
Pedro Holanda explains why lakehouse formats like Iceberg got so complicated: they began as plain files on object storage but quietly needed a database for transactions anyway. That observation is the founding argument for DuckLake.
Yeah, so the lake house is a different story than DuckDB. And the thing is, maybe we could start off like why lake houses, right? Like why do people even care about it? How does
a duck lake relate to DuckDB? Like how does that all come together?
So the main reason is that these lake house formats, they exist so people can exchange their database systems without having to do migrations, right? Like you don't really want to be stuck to one company because your data is on their proprietary formats or whatever. You want something that's open, that anyone could read a specification and implement it and then be able to interact with your data without you having to modify it. So there has been lake house solutions out there. That's not really something new, right? Like you had Iceberg, you had Delta. I think Iceberg maybe originated from Netflix even in 2017 or something like this. And their whole point of what they were doing was that your whole data should be able to be stored on S3. So just file formats, right? So the metadata, the way that you can traverse and figure out which data files you care, it's just JSONs and Avros. Of course. And your data is stored as Parquet files. What we realized is that at some points, they added a little database there. Because the whole point is that you had no databases. But it turns out that to have transactional control, so assets, you need to have a data set. Did need a database system…
Yeah, no, absolutely. And then you, or not you, Duck Lake, that is your baby, stores metadata in a SQL database and then data in Parquet, as you said. But then what does day-to-day work does that metadata design simplify? Like, how does that actually simplify? Because it does sound more complex to…
Oh, okay. So the simplicity is again in the way you actually fetch your data, right? Again, if we think about having all your metadata in files, you need to know which files to fetch. You need to read them. This is going to tell you which are the other files. And you end up like traversing a…
So everything's going quacky right now. Do you have to then be a DuckDB customer to then use DuckLake, the open source project?