Cory Doctorow
On Software Engineering Daily
“A radiologist who examines x-rays and gets a second opinion from the AI when the AI disagrees is doing their job at a higher degree of fidelity, and that stops people from dying of cancer. Versus the radiologist who sees nine-tenths of their colleagues fired and who then has to review 100 times as many x-rays that the AI has vetted and says are cancer-free. If the AI is wrong 3% of the time, they're probably going to miss all of it and then people are going to die. … That's the social arrangement of the machine, not the technical capabilities of it.”
Software Engineering Daily · Hands-On Engineering Podcasts · September 2026
Doctorow was explaining his book's distinction between a centaur, a human using a machine, and a reverse centaur, a human conscripted as the machine's peripheral. He had just invoked automation blindness, the well-documented finding that people cannot stay vigilant for rare failures, and cited TSA screeners missing most red-team weapons. His point is that the same tool produces opposite outcomes depending on who deploys it and why.
Transcript
this is a culture problem, right? I think that, you know, people who slop their PRs lack the discernment to distinguish useful code from useless code. And we have seen the free software ethic become an open source ethic, become a kind of open source as a career calling card ethic. And I think a lot of people are like, well, if I have 25,000 contributions to significant projects on GitHub, I can get a great job. Or I'll have bragging rights or something. I think the problem is they don't know that their code sucks. And I guess it's because they understand a lot about how code works and nothing about how code fails. They need to do something. You know, I run into this all the time because I'm not a software developer and I haven't written any code in a quintillion years. And yet I'm an Ubuntu user and like I get into trouble on my own by like copying and pasting from Quora or whatever when I'm like, I want this menu to work differently. I'm going to go recompile this piece of software or whatever. And I'm just sitting there going, make, make, install, please. And that's fine because if I like vibe coded a patch that fixed something for me and then it broke, I could just download like the official release and it'd be fine again. But if I vibe code a patch that breaks for 5% of users, of which I'm not one, because it breaks when you're using non-Roman script or breaks when you use like just one emoji that you never use or what have you, then you create a lot of headaches for the maintainers that you yourself can't perceive. And I don't think we can fix that just by scolding people. You know, I haven't really thought this through. So I'm thinking this through as I speak. So I think maybe what's happened is normally, if you're a bit of a dope and you've got an idea for a patch, that won't work. You write that patch, you submit it, the maintainer either gently or not so gently goes, look, dope, here's why your patch doesn't work. And maybe you work with them and actually you learn how to write good patches from them in the same way that like writing good bug reports or anything else, just it's like it's not a thing you're born knowing how to do. And what we've done is we've taken dopes and we've given the ability not to make one bad patch, but to make a thousand bad patches. And maybe what we do is we delete 999 of them and we say, look, dope, pick the one that you think is important, right? You want to fix something because you've got an itch you want to scratch. Something doesn't work that you think needs fixing. You want to be a part of this collective project. Why don't I, the maintainer, and you work together with your code assistant, if need be, to try and make the patch work the way we would with any other patch from someone who is just starting out. And we give them at least that much grace. And if they're not interested, if they are like, no, I just want to make a thousand patches and put it on my resume, then we show them the door. But like, we've had automated code gen tools forever, right? Like when I was teaching myself Apple Basic, my Apple II Plus did not have a debugger. And I read about it in like probably Byte magazine. I read about debuggers. And I wrote a debugger for the Apple II that was not very good. And I had a new automated tool that let me work with my code in a way that I couldn't before. And I was able to produce more code. And sometimes that meant that I could get it over my skis, but it also meant that I was able to fix my code faster. So I could write more code, but I could fix it faster. I could get it to run faster. And then because it was running faster, I could find the defects faster and I could fix them. And I think that used well, CodeGen tools that use the form of statistical inference we call AI can do the same thing. It's a matter of degree, but they have to be in the hands of someone who has discernment, particularly if they're working on a project that has other users. I think the place where vibe coding belongs is like you've got a home kit gadget and you've got a Raspberry Pi and you want to glue the two of them together and you vibe code an app and it works until it doesn't and you vibe code another one. That's fine. It's like the people who write visual basic or hypercard programs that like solve some tiny business need around their shop rather than hiring a programmer and trying to convey to the programmer what they're looking for. They just make the thing that they want and it doesn't fail very gracefully, but it works perfectly. It exactly scratches their particular edge. That's what vibe coding is for. It's not for submitting PRs to a big open source project that has a lot of people who rely on it and who are going to get right up the maintainer's butt when it stops breaking, when it stops working for the good reason that they rely on it.
I think that's also very good advice for folks in a corporate landscape. You know, the sitting down with someone one-on-one, helping them, working with them and developing personal rapport is particularly useful for coworkers, especially if you have a senior engineer helping a junior who's been much more AI trained and focused than them.
Yeah, I mean, people who actually work in software engineering know that like the major part of their job is not writing code, right? It's thinking and, you know, figuring out how the problem works and understanding how it works as part of a system. The difference between software engineering and coding is the difference between thinking about a system and thinking about some lines of code. And again, like coding is fun. I had the enormous privilege and pleasure of working with EFF's first ever staff technologist, who was a very playful coder named Seth Schoen. And Seth, he works on a lot of big open source projects where he was thinking things through and so on. But he also would write obfuscated C and he rewrote DCSS, the program that was illegal. It was a felony to distribute because it would decrypt DVDs. He rewrote it as functional Turing complete haikus. And that's incredible. Go look up the DECSS haiku. It's online. And eventually, and I'm not allowed to tell you he wrote it because he did not disclose he wrote it because it was technically a felony to write this poetry, which was his whole point, which is that the government had made it a felony to create literature and that this was a facial violation of the First Amendment. Go look it up. It's amazing. So writing code is incredible, but it's not software engineering. Software engineering, there was actually a really good piece in CACM, the communications EACM, that just came out that looks at the proportion of a software engineer's job that's writing code. It's just like, it's just not the major important part. And so LLMs, they let you do part of the job faster, but they don't let you do the rest of the job faster. And they create the danger that if you do that part faster and then try to debug it or QA it, right, do the code review for it, that because there's so much of it, it's going to be really hard to keep up with. There's this well-understood problem called automation blindness, which is that when a machine usually runs right and you're supposed to be vigilant for the very rare instances in which it runs wrong, when you're supposed to be the human in the loop, you can't maintain vigilance. Like just something about the human neurological apparatus seems incapable of maintaining vigilance for things that only occur occasionally, which is how it is that we've created in the TSA the water bottle spottingest motherfuckers the human race has ever produced, who are nevertheless like 95% of the time incapable of spotting the fake guns that red teams bring through the x-rays, right? Because they can't remain vigilant for these things that never happen. So if you're a programmer using AI to write code in a way that does not require superhuman vigilance, you know, an example might be in radiology, the difference between a radiologist who examines x-rays and gets a second opinion from the AI when the AI disagrees and says check that x-ray again, which is a thing where you're just doing your job, but at a higher degree of fidelity that stops people from dying of cancer, versus the radiologist who sees nine-tenths of their colleagues fired and who then has to review 100 times as many x-rays that the AI has vetted and says are cancer-free. And they have to make sure that the AI is right. And if the AI is wrong 3% of the time, they're probably going to miss all of it and then people are going to die. And, you know, that's not what. Machine does. That's who it does it for and who it does it to. That's the social arrangement of the machine, not the technical capabilities of it.
To me, that was probably the most impactful example of a reverse centaur versus a centaur itself. And I do want to talk to you, Ashley, about what you quote as dog shit Ugnit economics. But first, I have to ask, for the sake of the technical users, there are a lot of people who are trying to protest models or AIs as created today in terms of free speech. And you're a notable free speech advocate, and you mentioned this. Can you walk us through how is it that free speech laws protect things like the DECSS haiku and model generation for AIs?
Well, so the DCS haiku arguably is still a felony, unfortunately, because Section 1200 of the DMCA is very bad. And Judge, a very dumb judge who has done a bunch of terrible things in his career, including letting the Sacklers off, decided that the people who published 2600 magazine were not allowed to publish DECSS because it was not a literary work. So in the 90s, the NSA had classed, as you said, classed working encryption as a munition. And EFF and our fellow fighters for working encryption, privacy, and security raised a lot of arguments about why this was bad. The NSA said, all of these, this parade of horribles you proposed that will happen if we don't have working encryption, they're just hysterics. We will let you use a cipher called DES50, Defense Encryption Standard of 50 bits, that is so secure that no bad guy will ever break it, but not so secure that we can't break it. And so we'll catch all the bad guys. And we made a lot of arguments about the technical insufficiency of DES50. No one really cared. The NSA had hired like the largest plurality of top mathematicians from all the IVs and Big Ten schools for 50 years. People were like, who are you to say that this doesn't work? So then we proved it. John Gilmore, who was EFF's co-founder, who helped write Solaris design the SparkChip, write GCC, and founded the first ISP, as well as EFF, and also the Usenet alt hierarchy. He's quite a storied technologist. John built a computer. He made some custom ASICs. The computer was called DeepCrack. It cost a quarter million dollars. And it could brute force all of DES50 in two and a half hours. And we brought that into Cora. We brought that into Congress. And we said, look, if the mafia is willing to spend a quarter of a million dollars, they're going to be able to decrypt all the financial transactions in America, all the corporate secrets, all the individual information. Someday when we're pushing firmware updates to anti-lock breaking systems or people's pacemakers, that will be an open book to griefers and foreign spies and what have you. And again, they were like, yeah, that sounds stupid. Go away. So then we brought the lawyers in. And Cindy Cohen, my former boss, who's just retired from EFF and has written a wonderful memoir of this period. Cindy Cohen, called Privacy's Defender, I should say, Cindy Cohen, formulated a radical argument. She said the First Amendment protects expressive speech, and code is a form of expressive speech. And a lot of people at the time were like, I don't get it. How can code be expressive speech? But we represented a programmer called Daniel J. Bernstein. He was a graduate student at UC Berkeley. And he was publishing the source for a cipher that was stronger than DES50 and therefore a munition on Usenet, this early messaging system that predated the web. And we went to court on his behalf, and the judge agreed with us that this math that he was producing, this code he was producing, was a form of expressive speech and that the First Amendment protected it. And then we went to the appellate division and the appellate division agreed. And the NSA did not want to try their chances at the Supreme Court. And so since about 1993, it has been illegal to have encryption that works. And this is how we protect everything. I mean, everything. And it's because of this free speech tradition. And so when people say that we should ban certain code with a kind of scrutiny, so there's some speech that's unlawful, obviously, but the constitutionality of a law that bans certain speech is very fraught and requires that the ban be very narrowly construed and fall into a very small number of purposes, and that the speech itself be of a low degree of democratic importance. So restrictions on political speech, for example, are very hard to craft and in a way that comports with the Constitution, because one of the purposes of the First Amendment is to guarantee political discourse. And so this is one of the reasons that most so-called political disinformation and misinformation is probably illegal. Not all of it. I mean, stuff like vote next Wednesday when the vote is on Tuesday is illegal. Fraud, like deceiving people is illegal. Having a thing labeled as a deep fake, right? Where you're like, we're about to show you a deep fake of politician X saying some things that are a paraphrase of what we think he would do is legal because the First Amendment protects it, because the First Amendment is very, very delicate about restricting political speech. And one of the things that we're seeing right now is what happens when a government decides that it is not interested in protecting dissident political speech and how dangerous and frightening that is to the democratic project. And so there are outputs of AIs that are unlawful speech, but I would be reluctant to say that we should lower the degree of scrutiny we apply to restrictions on code itself, including code that embodies a model, not least because it's very hard to make that restriction stick. Like, what are you going to do? Like, make every Git server in the world comply with orders to take down the code that embodies the model, right? Like, how is that going to work?
It doesn't seem practical. I really wish we could spend more time talking about what goes into this. You also have content in the book about scraping and how that can be very, very good. But we do have to move towards the end of the interview. So let's talk about what comes next. What is dog shit unit economics in AI? And what do you think comes after a bubble burst? Or perhaps even why is the bubble going to burst?