Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
MH Magnus Hagander On Talking Postgres with Claire Giordano

“Our build farm, for example, has done just fine until AI agents showed up, and suddenly you need to optimize it in a whole different way, because they're not following the rules. The good old robots.txt that you'd put on a site to tell search engines what they're allowed to do: none of the AI systems care about that. So now you need to put more defenses in place.”

Talking Postgres with Claire Giordano · Hands-On Engineering Podcasts · November 2026

“Our build farm, for example, has done just fine until AI agents showed up, and suddenly you need to optimize it in a whole different way, because they're not following the rules. The good old robots.txt that you'd put on a site to tell search engines what they're allowed to do: none of the AI systems care about that. So now you need to put more defenses in place.” — Magnus Hagander, Talking Postgres with Claire Giordano

Context

Talking about the Postgres project's self-hosted infrastructure and what keeps its volunteer sysadmins busy, he said most recent work has gone into defending services from AI crawlers that overload them and ignore the rules sites publish.

Transcript

Talking Postgres with Claire Giordano Around 50:05 into the episode
Magnus Hagander

…uh uh kitania doing that so there's different people in different uh places i don't actually know who does the us ones these days but the point being like you should you should do what you're good at and uh graphical design or something that is not what i'm good at so i'm very glad to leave that to

Claire Giordano

…feel free to contact me um so it's a good talk and i will link to it thank you all right so uh we should probably switch gears from infrastructure except i don't know have there been any particularly exciting stories in the infrastructure space in the last couple of months that you want to share

Magnus Hagander

no not really a lot of the the things that are happening there are now are battling ai agents that are basically uh overloading pretty much everything on the entire internet and flooding everything everywhere so a lot of our our sort of some are site systems that are really just for postgres like our build farm for example has done just fine until ai agents showed up and suddenly you need to optimize it in a whole different way because they're not following the rules you put these the the good old robots.txt that you'd put on a site to tell search engines what they're allowed to do and what they're not allowed to do none of the ai systems care about that so now you need to to put more defenses in place for a bunch of these things and unfortunately that's what a lot of the work has been around uh recently

Claire Giordano

yeah i feel like that that's an evolving space though because obviously the ai agents can also help people right access information that was difficult to access previously and so um i just feel like we're in this like in-between state uh where you know they were it was like the wild west and it's…

Magnus Hagander

no no i think you're right in in general it's some some of this is just yeah nobody looked at the data before or very few people looked at the data before and more people are looking at the data now and in that regard it's probably a positive right once you manage to actually deliver it but as you…

Claire Giordano

okay so let's switch gears um from infrastructure then since i was fishing for like some dramatic story and doesn't sound like there is one at the top of your head right now um besides like you said the onslaught of ai agents um postgres the nonprofit which you've been involved with since 2008 so…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Talking Postgres with Claire Giordano