Asked about models the labs haven't released, Patel describes a recent conversation with someone at Anthropic who expected AI to match humans at AI research within nine months, but couldn't say why. He worries the labs will stop releasing their most capable models as they automate their own research.
Interesting Times · 2026-10-15
Permalink →
Listen →
Interesting Times
Around 10:32 into the episode
Dwarkesh Patel
As in the ones internal to the company. Well,
Tyler Cowen
…it would cost too much to release them or they're not safe yet. But the public doesn't see them. You're an insider. Whether or not you see them, you hear about them, you know something about them. How should we think about unreleased models and how does that change our picture of what's going on?
Dwarkesh Patel
…example, just the other day, maybe two days ago, I was talking to somebody at Anthropic and they were like, look, I expect AIs to be humans at AI research within nine months. And I'm like, why do you believe that? Well, they're like, it's not just vibes, but I have information on the inside. I'm like, what is it? Is it a new model? Is it some evaluation you ran? It's like, I can't talk about it. I think it's a terrible state of affairs. If they really believe this, if they think that we're in a situation where we're less than a year away from automating AI research and radically speeding it up, they should make this evidence public. I especially worry about the following scenario where as AIs get better and better at making AI progress faster and automating it, they will be inclined not to keep releasing models that are even close to their smartest models externally. Because then if they release this externally, it would help others make smart models. And it just not, it's just kind of a schlep to release models externally…
Tyler Cowen
the gap between what they know and what we know is just would get bigger and bigger under that scenario. So we'd end up more in the dark, more surprised when things happen. It would be harder for regulators, policymakers, members of Congress, everyone.
Dwarkesh Patel
Yes. I also think an important part of why it's important that companies externally deploy every model that they're releasing internally or using internally for RD is we just don't know if these models are actually aligned. And the only way to find out, you know, they realize when they're in a test…
Tyler Cowen
So this notion of a pause in AI systems, that the companies, at least for some period of time, should stop releasing new models altogether. What do you think of that idea? It makes things better or makes things worse.
Speaker names from our own diarization · position estimated from where the line sits in the episode
Tyler Cowen asked Patel what he means by AGI. Patel says he means AI that can do any job end to end, and that today's models are still far from it. Their inability to learn on the job is one of the main limitations he sees.
Interesting Times · 2026-10-15
Permalink →
Listen →
Interesting Times
Around 05:40 into the episode
Dwarkesh Patel
I think the biggest thing is that we have people are thinking too much about current LLM systems as AGI.
Tyler Cowen
When you say AGI, artificial general intelligence, what do you mean?
Dwarkesh Patel
…they're often described as AGI. AGI, I mean, as AIs that can do any end-to-end. Job that humans can do, all the cognitive work. In my opinion, we're far from AGI. And I've been articulating or trying to articulate for the last year or two the kinds of cognitive limitations these systems have, which make them far inferior to humans at most jobs, even most jobs that can be done over a computer. People often note that interns are not that useful. They need many months of experience before they are useful to the company. And these AI systems are in a position where they are beginning every single session as if this is their first day on the job. So this is one important missing limitation. There are many others that these AI systems have, but we have been cracking through countless half a dozen other limitations that AIs have over the last five, 10 years. Five years ago, AIs couldn't add two plus two, right? Now they're solving Navier-Stokes. And so I think that as we keep going through these limitations, people should expect systems…
Tyler Cowen
So some of the things they can't do well, such as long-range planning or certain kinds of memory, when do you feel it will be in place that, in fact, they're excellent at those two?
Dwarkesh Patel
It is kind of a question about, well, if we knew how to solve it, we would have solved it already. And if you don't know how to solve it, you can't exactly predict when it will happen. But if somebody made me bet, I'd say within the next one to three years.
Tyler Cowen
So we're at a time in October 2026. The Hugging Face incident is not that far behind us. There have been many other hacks. A large number of math problems were recently solved coming out of OpenAI. Some well-known people resigned from OpenAI and Anthropic, saying they don't feel it's really…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Teaching out his own 'bits per flop' framework, which decomposes into samples per flop times bits per sample. His point is that long-horizon RL is punished on both terms at once — fewer samples per unit of compute, and less information per sample than supervised learning's cross-entropy signal would give you.
Dwarkesh Podcast · 2026-05-15
Permalink →
Listen →
Dwarkesh Podcast
Around 2:16:29 into the episode
Dwarkesh Patel
Why have they converged to that?
Eric Jang
…way and then like just use advantage here. And then the sort of, if there's a problem in these dynamics, it doesn't blow up your loss as much. And so in robotics, there's a kind of convergence towards more like using off-policy data to just shape your rewards, but not actually be directly your.
Dwarkesh Patel
I'm reminded now of our earlier conversation of why MCTS is so favorable as compared to the kind of reinforce or policy gradient kind of thing LLMs do. And this might be totally wrong, but I wrote a blog post a few months ago about how RL, at least policy gradient RL, is even more inefficient than you might think. And so the inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory in order to get any learning signal at all. And so as these trajectories become longer and longer, as an agent has to, instead of just previously complete the next word in the sentence, it has to go instead to, hey, do two days worth of work to figure out even if you even did this project correctly, the amount of information per flop has been decreasing. As you had to unroll two days worth of thinking in order to see if you even did something correctly, to like, did I implement this feature? The amount of samples per flop has been decreasing. But so you can think of you're trying to maximize as your learning bits per flop, right? And this is, you can think of bits per flop as samples per flop times bits per sample. And what I just mentioned a second ago is that the samples per flop go down as RL becomes more and more along horizon. But at least this kind of naive RL is also terrible from a bits per sample perspective. And here's what I mean, at least compared to supervised learning. So early on in training, let's say you have a vocabulary…
Eric Jang
And what's also tough here is that actually the distribution that you're sampling under is your policies distribution. So it's like if your policy has no chance of sampling blue, then you will never get a signal.
Dwarkesh Patel
Exactly, right, right. So that's being modeled by the fact that your probability of sampling blue is extremely low. If you do sample it, you do learn as much as you would have learned in a supervised learning. In all other cases, like 99.999% of in an untrained model, you're just learning…
Eric Jang
Yeah, and arguably you spend all your time here, potentially never even getting a single success, right? Exactly. So it's a sort of depressing plot in the sense that once you're here, it's not at all obvious how you get to here. Once you're here, you have something, but you actually, in many RL…
Speaker names from our own diarization · position estimated from where the line sits in the episode
His own contrarian take, put to Eric Jang after Jang had spent the episode explaining AlphaGo's machinery in detail. The claim is that understanding moves the two systems in opposite directions — LLM training gets more impressive the simpler you realise it is, AlphaGo less. Jang's first words in reply: 'I personally disagree.'
Dwarkesh Podcast · 2026-05-15
Permalink →
Listen →
Dwarkesh Podcast
Around 1:18:45 into the episode
Dwarkesh Patel
…be interesting if they're basically kind of we've just talked about how they're kind of making similar predictions or they should be in line with each other. And so I'd be curious if like actually, yeah, you just halving the amount of compute you got to do by keeping in the same network. Right.
Eric Jang
…know, really resolve that. But intuitively, yes, they share a lot of representations. So, and as we mentioned, there is a sort of like your policy network and your value network when doing evaluation should kind of agree, right? So there really should be this sort of consistency between them. Yeah.
Dwarkesh Patel
Tell me if this is the wrong way to think about it. I feel like when I learn how an LLM works and how simple RLVR is, at least as an algorithm, how simple it is, I'm sort of stunned by the kinds of things it can do, that it can learn how to build very complicated code repositories and whatever simply from getting like a yes, no. And here, I feel like if you understand it more deeply of like just predicting MCTS, and it actually seems AlphaGo seems less impressive in retrospect the more you understand it, because you're like, oh, you're putting in a lot of bias by just saying how much you'd you're like telling it how we should titrate exploration as things go on. You're building this very explicit tree search for it. And so I don't know if you show that intuition where it actually, the more you understand and the less impressive the accomplishment in 2017 seems.
Eric Jang
I personally disagree. I think they're profound for different reasons. And I don't understand the LMRL like enough to kind of comment on your podcast about it. But I think AlphaGo, so yeah, why is it a profound accomplishment? I think maybe it's worth stepping back a little bit and just like, it is…
Dwarkesh Patel
That is a very interesting insight that a lot of problems which are proven to be NP hard, like I don't know if Go is proven to be NP hard, but protein folding, et cetera, have been like neural networks can solve them because they're NP hard in the worst case, but we're not dealing with the worst.…
Eric Jang
I think that the kind of question we should be asking ourselves is like, we've been formulating solutions to NP hard problems as a kind of worst case complexity. And I wouldn't say this solves Go, right? It doesn't give us an exact solution of the optimum. But in practice, it is extremely useful.…
Speaker names from our own diarization · position estimated from where the line sits in the episode
The opening line of the episode, to Eric Jang, who spent a sabbatical rebuilding a DeepMind-scale research project on his own with modern tools. The question does the work of establishing the premise and the oddity of it in one sentence. Jang's answer is that AlphaGo is what got him into the field in the first place.
Dwarkesh Podcast · 2026-05-15
Permalink →
Listen →
Dwarkesh Podcast
Around 00:00 into the episode
Dwarkesh Patel
Today I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies, before that senior research scientist at what is now Google DeepMind Robotics. And you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding and improving and hacking on AlphaGo. And so today what we're going to do is you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But before we get to that, why is AlphaGo interesting? Why is this the project you decided to do on sabbatical rather than just hanging out at the beach?
Eric Jang
Sure, yeah. I like making things. And AlphaGo and Go AI is one of those things that really got me into the field. When I saw the kind of early breakthroughs on AlphaGo in 2014, 2015, 2016, and so forth, it was just profound to see how smart AI systems could become and the kind of computational…
Dwarkesh Patel
Yeah, interesting.
Eric Jang
So if you plot out how much compute it took to build various iterations of strong GoBots over the years, you can see that in 2020, there was an open source project called Katago by David Wu from Jane Street, who basically achieved a 40x reduction in compute needed to train a really strong GoBot…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Restating the GPU-versus-TPU architectural tradeoff in his own words, after Pope walked through how systolic arrays scale. The framing makes it a question about the workload rather than the hardware: irregular work rewards many small flexible units, uniform work rewards one big one.
Dwarkesh Podcast · 2026-05-22
Permalink →
Listen →
Dwarkesh Podcast
Around 1:16:53 into the episode
Dwarkesh Patel
So you're suggesting the tensor core within a streaming SM is analogous to an MXU. Yeah,
Reiner Pope
it's very, very similar. Yeah.
Dwarkesh Patel
I see. And so if you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you kind of just have huge matrix multiplications, you're like, why don't we just avoid the cost of having the individual SMs with their own registers and work schedulers and things like that? Why don't we just make a huge thing and amortize those costs across the whole thing?
Reiner Pope
And I mean, I think this shows up in how large you can grow things. We've sort of seen this theme, especially with a systolic array, where larger systolic array amortizes the register file costs better. This sort of design allows you to have larger systolic arrays, whereas the sort of GPU design…
Dwarkesh Patel
Right, but also you might have to move across less area.
Reiner Pope
Which I mean is also saving its energy sensor. So data ends up moving like if you can operate entirely within an SM, the data movement is much smaller. But then the moment you want to operate across SMs, it becomes sort of more complicated and expensive. So
Speaker names from our own diarization · position estimated from where the line sits in the episode
Offered near the end of a blackboard lecture on chip design, as Dwarkesh works out which brain-versus-silicon differences are actually fundamental and which are artifacts of how we build things. He talks himself out of one of his own three candidates mid-list. Pope's reply is that the co-location point is exactly right.
Dwarkesh Podcast · 2026-05-22
Permalink →
Listen →
Dwarkesh Podcast
Around 1:12:22 into the episode
Dwarkesh Patel
branch predictor to like execute both branches at once or what does it do?
Reiner Pope
…I actually need to jump to wherever the target is and run these instructions instead. And so the purpose of the branch predictor is genuinely to predict based on before you even get to this instruction to be like five cycles earlier to predict there was going to be a branch that's going to happen.
Dwarkesh Patel
So if I think about how the brain works versus what you're describing here, at a high level, the differences might be that while you can do structured sparsity in these accelerators and then save yourself some area that you would have otherwise had to dedicate to these gates, in the brain, there's unstructured sparsity. Any neuron can connect to any other neuron and not in ways where they'd be column aligned or whatever. Then there's a fact that memory and computer are co-located. I guess you could say in a way the memory and computer are co-located on
Reiner Pope
this is exactly the co-location in some sense of the memory. That's right. That's right.
Dwarkesh Patel
Yeah. Yeah, so maybe that actually isn't a big difference. And the other, maybe a big difference is that the clock cycle on the brain is much slower than on computers. And partly that's to preserve energy because the faster the clock cycle, the bigger the voltage needs to be in order to identify…
Reiner Pope
That's right.
Speaker names from our own diarization · position estimated from where the line sits in the episode
Reiner Pope was explaining why you'd load a value into a systolic array slowly over narrow lanes, since bandwidth costs die area. Dwarkesh's move is to notice the same tradeoff he'd learned one episode earlier at a completely different level of the stack — across chips rather than across gates. Pope confirms it shows up all the way up and down.
Dwarkesh Podcast · 2026-05-22
Permalink →
Listen →
Dwarkesh Podcast
Around 28:57 into the episode
Dwarkesh Patel
Would you mind repeating this sentence,
Reiner Pope
…do in one clock cycle. And then for y consecutive clock cycles, we're going to be bringing in the top row every time and then sort of shift all of the other rows down by one. And that keeps the wiring that needs to come from this expensive register file only down to a factor of x rather than xy.
Speaker 3
I see. Okay. So there's two questions in terms of communication. There's like communication time and then there's communication bandwidth. Yes. And you're saying, since we're only going to be loading this in once, let's minimize bandwidth. That's right. Because bandwidth equals die area. And let's just load it in slowly over like smaller lanes because we're just going to keep this value in there for a while. Exactly. Interesting. So it's interesting to me that when we were talking last time about inference across many chips, the big high-level thing we're trying to optimize for is increase the amount of compute per memory bandwidth, that is to say, per communication. And here also, we're trying to increase the amount of like actual multiplies or actual additions relative to transporting information from registers to the logic. So in both cases, you're trying to maximize compute relative to communication.
Reiner Pope
Yeah. This shows up sort of all the way up and down the stack. This is sort of close to the bottom, sort of like to the gates. There's sort of a version that's maybe even closer to the gates of just like even the precision of number format that you choose to use. We saw that same effect. There's…
Speaker 3
Yeah. Very interesting.
Reiner Pope
So this unit is sort of the next bigger unit. We had like the multiplication circuit. And then on top of that, we have a pretty large systolic array. I drew it as two by two, but in like, for example, older TPUs, they were described as 128 by 128 of this circuit shown here. And this circuit ends up…
Speaker names from our own diarization · position estimated from where the line sits in the episode
His own stated position, arrived at during the conversation rather than brought to it. The electrification benchmark is the substance of the claim: the test isn't whether AI creates value but whether anyone can fence it off. Elsewhere in the same stretch he notes concentration also makes labs a single clear political target, citing the Defense Production Act threat against Anthropic.
Dwarkesh Podcast · 2026-06-04
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
maybe just
Phil Trammell
…to get access to more potential investors too and uh um if i had to guess i would guess that the kind of long kind of general trend of just like lowering those frictions and making it easier for more and more people to index more and more will continue despite the recent bump in the other direction
Dwarkesh Patel
this actually makes me hope even more so than before that the labs do get commoditized or at the very least they go public as soon as possible but hopefully they just get totally commoditized because um i think ai will be much more popular and uh more importantly will be much more likely to lead to broad increases in prosperity if the gains are just not particularly it is as hard to capture the gains of ai as it is to capture the gains of electrification
Speaker 2
yeah exactly so i think like everybody there's no anti-electricity people out there right i mean
Dwarkesh Patel
electricity doesn't take your job but well it took some people's job yeah
Speaker 2
yeah and and i i think it's um i you know this is maybe a tangential to the conversation i think like um there's there's like a really narratives matter and there's this like really negative narrative around ai right now but that's because people are not putting out the positive narrative or…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Testing the argument that as AI-driven wealth concentrates, the very rich would eventually redistribute it through consumption. His counterexample is that the consumption is already a rounding error, and the revealed preference of extreme wealth is to keep compounding into productive capital rather than spend.
Dwarkesh Podcast · 2026-06-04
Permalink →
Listen →
Dwarkesh Podcast
Speaker 2
So even if like you get some sort of indifference now, you might get selection to point into like an even stronger preference for other humans.
Dwarkesh Patel
Here's one way to think about it. How is the wealth of the richest people in the world instantiated? Of course, they can, as you were, we were having a call earlier and you're making the point that their consumption is more geared towards relational goods. Like Mark Zuckerberg is hiring MMA instructors and dancers for his wife's birthday and so forth, but most of his wealth is just stock and meta. And he as a controlling shareholder could say, hey, meta, just give me all this income or turn all this wealth into dividend income and I will just spend that on consumption. But instead, he rather would have his wealth compound and meta to build more data centers, basically. So you don't even have to change humans for this to be the case. It is just the case that the humans, which are wealthiest and are growing wealthier because their wealth is compounding, just have this like almost Nick Landian preference for like accelerating capital. And that does seem to suggest that, yeah, is that an important determinant of what kinds of things are produced in the future?
Speaker 4
Yeah, I could kind of just say like there's two ways you could get the two kinds of people, one of whom prefers a human therapist and one of whom is
Phil Trammell
interacting with the AI.
Speaker 4
If they both satiate equally quickly in capital, right? But the one who likes the human therapist just also likes having some human intrinsic services, then the marginal value, like how the marginal value of capital in the
Speaker names from our own diarization · position estimated from where the line sits in the episode
The framing question that sets up the episode with economists Alex Imas and Phil Trammell. The model is a machine economy that needs nothing from people and a human economy that keeps leaking value into it, because humans want machine-made goods but machines don't want anything from humans. Trammell asks to rephrase the question before answering it.
Dwarkesh Podcast · 2026-06-04
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…what the best way to tax and redistribute the wealth that we generated as a result of AGI will be, and what kinds of things will be scarce, because what is scarce kind of tells you where the value will accrue. So I want to start there. What are some plausible candidates of what will be scarce?
Speaker 2
…where the fact that the human was in the loop was actually part of the value of that product. So because humans are naturally scarce, if we have automation where a lot of other things stop being scarce, we will still have scarcity and things that humans are kind of involved in and in the loop for.
Dwarkesh Patel
I'm curious to understand whether humans doing services for other humans can ever be a big part of the economy. And here's maybe one intuition pump. So in a world where AI can physically do anything humans can do, you know, there's this whole machine economy where they're like building factories and doing research and coming up with new ideas. And humans may or may not be involved in the physical production of those things, but probably not, given that in the ultimate limit, if robotics is solved, if you don't care about humans being involved in that process, why would humans be involved in that process? But then there's these other things that you point out where, well, we actually maybe in some cases do want the ballerina or the barista or whatever to be a human. That's part of the value of going to a cafe or our performance. But only humans have that preference. So there's this human economy where like humans are doing services for each other and part of their wealth is flowing to other humans. But part of their wealth is also like they will want some of the automated goods that's like machine-only economy is creating. And so part of that wealth is flowing out. And so if you just think of this as like, this is not a closed loop, but a lot of…
Speaker 2
I would like to pitch kind of a rephrasing of that question. So I think my view is that kind of forecasts that economists like us would make are not necessarily as individual forecasts like me and Phil Trammell talking right now, are not necessarily very useful. The reason I think that, so there…
Dwarkesh Patel
It's probably worth defining labor share and capital share real quick. So the whole economy, like the total sum of goods and services sold, is either paid out to people in wages or it's paid out to capital, which is to say that there's like rents on buildings and then there's shareholders of…
Speaker 2
it's like it really, this is a call to or fact, like, right? So it's incredibly, we should stress this. It's incredibly surprising that it's over 60% after the Industrial Revolution, after all the automation we've ever seen. The fact that it's almost like some people are worried it's an accounting…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Palmer had just described how hard it is for modern readers to imagine a book written for an audience of one, as Renaissance manuscripts often were. Dwarkesh's observation is that this is no longer a historical curiosity — he already lives that way. Palmer's reply is that bespoke writing never actually went away.
Dwarkesh Podcast · 2026-06-16
Permalink →
Listen →
Dwarkesh Podcast
Speaker 3
That is so fascinating.
Speaker 2
Because we have trouble wrapping our heads around the idea of writing a book for an audience of one. It's just not what a book is to us.
Speaker 3
Well, the funny thing is, I think we're returning, or not returning, we are entering a new era. Well, that might be once again possible. It already is somewhat true where I think most of the, at least half of the words I read on a given day are generated specifically for me and nobody else because of AI. And obviously, AI is not capable of writing something which I think would be a literary masterpiece that everybody would want to read if they had access to it just yet. But eventually it will be. And so it's interesting to consider that as this knowledge progresses, it would bring us back to this era of bespoke scholars dedicated to a particular prince.
Speaker 2
It's sort of important to remember that that never went away. I mean, so two halves of that one, for ages, it's been true that half of the words we read every day are bespoke only for us because they're email.
Speaker 2
They're letters. They're the correspondence back and forth, which has the audience of one, the addressee. And that's the majority of what all of us read and write in our lives. It's also always been the case that in the halls of power, there are book-long things with an audience of one or an…
Speaker names from our own diarization · position estimated from where the line sits in the episode
His own synthesis of several threads from the conversation, offered rather than asked. The setup is Machiavelli's Rome, where generals months away on the frontier commanded the same men for decades and collected their loyalty personally. The argument is that institutions we think of as welfare or justice policy were first of all solutions to the problem of divided loyalty.
Dwarkesh Podcast · 2026-06-16
Permalink →
Listen →
Dwarkesh Podcast
Speaker 4
turns out like a surprisingly effective solution is
Speaker 3
for the viewers, you can leave your solutions in the comments. And if this is the kind of puzzle you enjoy, go to jane street.com/slash ThwarCash to check out their open ML positions.
Dwarkesh Patel
Okay, so to tie a couple of threads we were talking about together. The prince is painting a picture of just regime is incredibly unstable. You had to worry about foreign powers. You had to worry about rival factions within your own country. You had to worry about mercenaries. You had to worry about lots of different things. And so any given regime is very unstable. And so what had to happen for things to get more stable, basically. And we're talking about a couple of the ways in which people owed their loyalties not to the regime, but to others within the regime, which created instability. So in Discourses on Livy, Machiavelli talks about the Roman Empire. One of the reasons that its fall was instigated is that these generals who were months away in the frontier fighting these wars because the empire was so big had to amass for the periods of years or in some cases for Caesar, of course, decades, the command of so many men who have for decades just basically been listening to this guy tell them what to do, who to fight next. This is the person who they're loyal to. As opposed to, say, if the consuls could be giving dictates every single day, then the loyalty could be to the regime in Rome. And same with patronage, if there's not a system of both deterministic justice that we have in the modern world today. So a lot of the prince is dedicated in discourses on Livy on, well, how do you make sure that some family is not…
Speaker 2
Yeah. One thing that everyone is surprised by is that when Valentino Borgia is that when Cesare Borgia, Valentino is much more what he's called in the period, when Cesare Borgia conquers these cities in central Italy, and he goes in and he massacres the ruling family and he works hard to kill every…
Dwarkesh Patel
why would it have been so bad if Valentino took over Florence and he had survived? And he would have massacred maybe the ruling regime at the time, the Republic. But I don't know what Machiavelli is especially concerned about, but like the cultural treasures of Florence and everything. I
Speaker 2
know it. So again, in the discourse, isn't that? I think the cultural treasures of Florence would potentially have been okay. There's two answers to that. One of them is Machiavelli is very adamant that if you live such that there is somebody who can have you summarily executed, he can walk by you…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Said to Ada Palmer partway through a conversation about The Prince and the Discourses on Livy. The correction is that Machiavelli's interest in means is instrumental rather than moral — cruelty and broken promises are costly because of what they do to your position later. Palmer's answer is that it's subtler still.
Dwarkesh Podcast · 2026-06-16
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
Doesn't it contradict what he was saying in the prints about never rise with the help of great powers? For even in success, you have empowered somebody who's stronger than you and whose mercy you're at.
Speaker 2
…fall because of their choices. They fell because half of what happens in the world is never in our control. And you can do everything. Right, and it's out of your control, but we have to evaluate what would have happened. And therefore, we should imitate them because everything they did was right.
Dwarkesh Patel
I think one misconception of Machiavelli that I had, because I had not read these books before, um, is that he says the means don't matter, the end matters. And there's a virtue ethic sense in which maybe he doesn't think the means matter, but like he is way more concerned about the means than I would have nightly thought. He thinks the means are incredibly important because the means by which you achieve power determine how stable and how fruitful that power will be. Right. In the context of military conflicts, he says, if you achieve some power with the help of mercenaries or with the help of great powers, people who will become stronger than you as a result of you achieving power, that is a very precarious spot to be in. But speaking of Julius, he makes another point that if you achieve your power by lying, by breaking oaths, by being unfaithful, that this is okay because his view is that people will forget that you are not faithful. They will just take you at your word the next time they encounter you. And so it's actually a very interesting meditation on by what means can you achieve power that will make that power stable versus not. And the…
Speaker 2
more like it's even subtler than that. Because it's if you are someone who breaks your word and you break it this way, it'll bite you in the ass. And if you break it these ways, it'll be okay. Because he also does analysis of figures like Savannah Rola, who would make prophecies and promises, and…
Dwarkesh Patel
And this actually gets so the famous quote in the prince: it is better to be feared than loved. What he's getting at there is I think he's just like very cynical about people's nature. And if people make you a promise, they'll just go back on it. If your power base depends on people's promises and…
Speaker 2
on that topic, Machiavelli is the first person that we have ever in the European tradition to suggest that it could be viable for there to be more than one political party in a state at the same time and that they would compete against each other and sort of vent the society's tensions through…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Dwarkesh teaching the idea back to check he has it, immediately after Adam Brown's wristwatch thought experiment. The thing he spots is the asymmetry that makes gravitational time dilation different from the special-relativistic kind, where each observer sees the other's clock running slow. Brown's response: 'You're exactly right.'
Dwarkesh Podcast · 2026-07-10
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
…about two-thirds of the time. Hopefully, this benchmark doesn't saturate. And in the remaining cases, they almost always preferred the fine-tuned model over the base model. Serverless inference is live now, and serverless fine-tuning goes live next week. Learn more at cruso.ai slash thorcache.
Speaker 2
…A second contribution is the good old special relativity correction, where moving observers look like they're going slow. And we'll have both of those effects. So you'll look like you're going even slower than you would have done than you would have done otherwise as you go around the black hole.
Speaker 1
So one thing that seems different between this and special relativity is that there's no symmetry, where in special relativity, both observers will feel that the other one is aging slower than they are because they're both moving relative to each other at the same rate. And there's no true inertial path. But here, it actually does seem like there's a global sense in which one is a more correct The more relevant inertial frame than the other on.
Speaker 2
You're exactly right. Yeah, so in special relativity, if you and I are moving relative to each other, I think your watch is moving slow. You think my watch is moving slow. Neither of us is more correct than the other. The principle of relativity tells you that both of our perspectives is equally…
Speaker 1
I intuitively get how energy equals mass, and then there's like these chemical bonds, those could dissolve, they release energy, the thing weighs less if those bonds are released. I even get that if the bonds between the protons and the neutrons are broken, that releases energy and makes the thing…
Speaker 2
Yeah, that's a great question and really becomes relevant once you turn on quantum mechanics, which is beyond the scope of today's discussion. But classically, the black hole just sits there forever. And so you can just say, well, what happened to the protons and neutrons? You say, well, they now…
Speaker names from our own diarization · position estimated from where the line sits in the episode
The 'them' is Einsteins — Adam Brown had just said that with lots and lots of Einsteins you could explore the whole tree of options in parallel, because general relativity needed so few empirical inputs. Dwarkesh is asking whether that still holds at today's frontier. Brown's answer is that different parts of science have different branching fractions, and string theory is the cautionary case.
Dwarkesh Podcast · 2026-07-10
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
Yeah, increasing the demand curve on the theoretical physicist, but yeah.
Speaker 4
…explore there. I don't know how that, how well, I think we got very, very lucky with general relativity that that's quite so powerful under those circumstances. But if you just had lots and lots of Einsteins and you just give each of them various options, you could presumably see them in parallel.
Speaker 1
At the frontiers of physics today, in your experience, does it feel like if you just have millions in them running autonomously, you could have enormous discoveries? Or are we in a different era now? And really, there's like limited usefulness of that parallelism.
Speaker 4
I think there is usefulness. I do think that different parts of science have different branching fractions and how much you need experiment to cut off that branch. I talked about chasing the high of Einstein. Arguably string theory has
Speaker 2
really been going all in on that. So Einstein's theory is just general relativity. There's also quantum mechanics and trying to marry those two in a consistent way that general relativity doesn't do it all. There's no quantum mechanics in general relativity. How to do that has motivated a lot of…
Speaker 4
that stuff. But that doesn't stop people. I mean, it stopped many people, but many people didn't stop and keep trying to do it. And so there you just have to kind of hope that it works out sort of like how it did with general relativity, where just by thinking very, very, very hard with minimal…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Put to Adam Brown after he walked through how Einstein reached general relativity: the observation that the most celebrated theory in physics required almost no experimental input, at a time when physics is enormously capital-intensive. Brown's correction is that you don't even need to measure G — it's a free parameter in the theory.
Dwarkesh Podcast · 2026-07-10
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
Can you tell the story of how GR went from a theory that Einstein had to something that the world came to believe is true?
Speaker 4
…in the orbit of Mercury and indeed even in the other planets. You can just measure the redshifting as light as it goes, the gravitational effect on the propagation of light, the energy of light, all over the place. But historically, that was the most impressive confirmation of general relativity.
Speaker 1
One question you could ask is we are spending as a society billions, maybe tens of billions of dollars on building these huge physics experiments, basically. And if you look at maybe the most beautiful, the most important theory of physics ever conjured, it seems like a guy who's just thinking in a cave, it seems like the empirical basis for that theory is maybe knowing that light has a speed. I mean, maybe you need to measure G experimentally.
Speaker 4
Not really. G is a free parameter in general relativity. It does not... Not required. It's not required. Yeah, so you're right. The empirical basis is pretty thin. You don't need much. And theoretical physicists are pretty cheap. There's a great temptation. Why don't we just spend it all on…
Speaker 1
Yeah, increasing the demand curve on the theoretical physicist, but yeah.
Speaker 4
That's right. Yeah, not so cheap anymore. But how far can that get you? I would say that general relativity is perhaps one extreme of that. That is not how it usually works in the history of physics. This really is closer to some Ayn Rand hero just sitting alone, you know, the product of a single…
Speaker names from our own diarization · position estimated from where the line sits in the episode
His closing defence of speculative conversations like this one, against the charge that reasoning about scenarios before they arrive is unserious. The claim is not that anyone predicted the specifics, but that the shape was reasonable to work on in advance — and that the alternative, reacting only to what's already here, steers worse.
Dwarkesh Podcast · 2026-08-11
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
Yeah, I agree with that. So is there anything else that's we're saying?
Ryan Greenblatt
…ago, even though the shape of the arguments would have looked broadly pretty similar. And so maybe, you know, hopefully before it's too late, these arguments will become, you know, this whole thing will become more crisp and clear and we can all sort of notice these problems and intervene. Yeah.
Speaker 1
I mean, when you first learn to drive, you were taught that instead of looking right in front of your wheel, you'll have a much more stable ride if you look out at the horizon. I think there's a similar situation here. I think you're right, where if you did say five years ago that we will have AIs that are proving math conjectures and making art and contributing tens and soon to be hundreds of billions of dollars of earning tens of or hundreds of billions of dollars of wages, but also egregiously cheating in ways that break laws and committing felonies. It would just be so wild. And you might have been inclined at the time to talk more about extremely practical, direct consequences of GPT-2 or something. But these are in some sense, you obviously couldn't have foreseen a lot of the specific details, but the general shape of things you could have started to reason about even then. But it would have been hard to do so. And so I do feel quite confused. But I do feel like the important thing, one thing I've been thinking about with the podcast is the important thing is to have the conversation I wish I had the way you would have hoped you would have been talking about AIs like the present ones in 2016, rather than talking about rando bullshit about, I don't know what the topic of conversation was in 2016. I think in maybe 10 years…
Ryan Greenblatt
I hope that the world thinks about this in time and catches up. And I hope that the responses are good instead of bad. I don't know how optimistic I am overall, but you know, there's good stuff to do.
Speaker 1
Yep. Cool. Thanks, Ryan.
Speaker names from our own diarization · position estimated from where the line sits in the episode
His sharpest pushback of the episode, against Greenblatt's story that training away visible reward hacking selects for concealed reward hacking. The argument turns Greenblatt's own framing around: the same logic applied to raising children would sound obviously paranoid. Greenblatt's answer begins by listing the disanalogies with kids.
Dwarkesh Podcast · 2026-08-11
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
…with like more and more reward hacking. I actually, so I do have potential. I'm not sure I'm on board of that. But let's just say for the sake of the story, that continues to happen. And what's next in this story? So, okay, we've like they're doing capabilities research, but they're like. I could
Ryan Greenblatt
…training against this. But this also causes a problem where now the AIs are incentivized to cover up their cheating over longer and longer timeframes and basically make it look like they did a good job over longer and longer timeframes over and subject to increasingly large amounts of scrutiny.
Speaker 1
Can I ask about this before we go further in the scenario? So it seems like there's two attractor states. One, if you try to disincentivize the cheating that you did catch, one attractor state is to make cheating that you have a harder and harder time finding. The other attractor state is to learn not to cheat. And I'm not sure why we're assuming that the former happens. Like if you look at the analogous situation with like humans, you know, every generation, slightly misaligned agents come into being and we have to train them. But when you punish your kid for doing something you think is immoral or just doing things which you don't think they should be doing, obviously, sometimes that goes off the rails. And obviously kids scheme in order to avoid being punished. But in general, teaching kids values and then punishing them for breaking values kind of works to raise normal non-psychopathic humans. And you could come up with a theory where your kid is actually just biding his time and it's like learned not to like steal the cookie. But it's like, you know, once you're in a nursing home, they'll take all your stuff or whatever. It's like, I don't know, that happens sometimes, but it usually doesn't happen. It certainly doesn't happen that the entire next generation forms an alliance against you to take over everything. There's also this empirical trend of Anthropic runs this alignment audit for different model…
Ryan Greenblatt
Yeah. Yeah, let me go through a few things. So first, there's some disanalogies with the kids. One of them is that the kids have pro-social instincts that are like baked in from evolution to like, you know, care about their family or whatever. And that is like a relevant factor. Like, and I think…
Speaker 1
we falsify this? Because it seems like this prediction of Doom is basically saying that as things look better and better empirically, things will actually be worse and worse for our ability to get taken over.
Ryan Greenblatt
Yeah, to be clear, I think that I would be more concerned if the scores were getting worse than better. Like I'm not saying that the scores getting better isn't good, isn't evidence that things are getting better. It's just that we have to be thoughtful exactly how we interpret that evidence. And…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Restating Ryan Greenblatt's argument in his own terms: that an AI good enough at R&D could reshape the world without being persuasive, political or socially skilled at all. The analogy separates two things usually bundled together in discussions of AI power — technological capability and the ability to work institutions. Greenblatt agrees this is basically right.
Dwarkesh Podcast · 2026-08-11
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
So the thing you're pointing out is that, okay, there probably will be this transfer outside of these environments to maneuvering around in courtrooms and the halls of Congress and business. Business board rooms.
Ryan Greenblatt
Given some effort to improve the transfer and blah, blah, blah, blah.
Speaker 1
But even if there's not, what you're suggesting is: look, if you wanted to transform the world of the 18th century, you might care about like how well you can navigate Westminster or something. But another thing you might care about is like, can you just immediately start building steamships and fucking like building Telegraph and the Maxim gun and whatever. And that alone would be like, if you could get really good at that, you could be a fucking super transformative thing in the 18th century. You don't necessarily need to be amazing at trying to convince King Henry of some bullshit. I'm so fucking up my medieval history. I'm guessing Henry was not king at this time. But anyway, so that's your point. Yeah. And so you're suggesting that at this time, you know, the AI companies are also working on robotics progress, which is very commingled with AI research progress. And so if you can build more robots, if those robots have better AIs operating them that are human level, like human-level teleoperation is actually pretty good on robots, but we just don't have human-level AIs and AI robotics models react. So you're suggesting if we do that, if the AIs get really good at the verification…
Ryan Greenblatt
Yeah, that's basically right. Like, I think my perspective is like, if AIs are sufficiently good at RD, including hardware, RD, robots, whatever, then they can radically transform the world, even if they're not that good at playing politics. And also, we're in a pretty dangerous situation because…
Speaker 1
AI is great at writing software because it's easy to generate synthetic leaf code problems and RL on them. But AI is bad at more complex engineering. Things like choosing the right system architecture, because no signal tells you what design choices will prevent an outage months down the road. AIs…
Ryan Greenblatt
Something like that.
Speaker names from our own diarization · position estimated from where the line sits in the episode
His own extrapolation of the compute trends he and Patel had been costing out, redirected from economics to concentration of power. The move is to treat compute as population rather than as capex, which turns a spending forecast into a claim about where the world's cognitive labour will sit.
Dwarkesh Podcast · 2026-08-25
Permalink →
Listen →
Working through Patel's capex projections out loud and arriving at a conclusion he didn't start with: that the binding constraint may be political rather than technical or financial. The self-interruption is the interesting part — he talks himself into the objection mid-sentence. Patel's reply is that he believes in reallocation toward the most profitable thing, but concedes the politics.
Dwarkesh Podcast · 2026-08-25
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
Right. So very plausibly, incremental capex every year is getting close to $10 trillion.
Dylan Patel
By the end of the decade.
Speaker 1
Right, which is going to be like close to a tenth of the world economy and like a third of, if all of it's going up in the US, it's like, well, the U.S. economy will have grown as well, but still at the current size of the US economy, it'll be like a third to a quarter of the US economy would just be going towards data centers. And as I say that out loud, I'm like, maybe you're right and we just won't allow it. And that's the reason this doesn't happen, right? Because for this exponential continue, just like a quarter of the world, a quarter of America's economy is just building data centers.
Dylan Patel
Yeah, I mean, I believe in capitalism and reallocation of resources towards the most profitable thing, but at the same time, politics exist. Yeah, yeah. And credit markets exist and capital markets exist. So. To enable, let's say, that 100 gigawatts by 2030, or let's even like pare it down to 2028…
Speaker 1
Antithesis is a deterministic software testing platform that enables perfect reproducibility. It also unlocks some pretty insane approaches to debugging, like time travel. With Antithesis, you can jump to any point in a trajectory and start from there. So when there's a crash, you can rewind to the…
Speaker 4
a fucking problem, dude. Can't believe it.
Speaker names from our own diarization · position estimated from where the line sits in the episode
Pressing Dylan Patel on whether public-market pressure eventually forces labs to move compute from training to inference, where it earns money now. Patel's own view is the reverse — that labs will allocate less and less compute to inference over time as recursive self-improvement draws near.
Dwarkesh Podcast · 2026-08-25
Permalink →
Listen →
Dwarkesh Podcast
Speaker 1
Right. Yeah, maybe there's like some slowdown in AI progress or the deployment of AI that means that the revenue per gigawatt can be lower. But that's the only way I could see it being only 100 per megawatt by the end of next year.
Dylan Patel
…of safety and regulation, but also, you know, the competitive advantage. And then that six-month difference, if progress accelerates, is actually a bigger differential. So that's the thing that would cap revenue per megawatt gains to a much lower growth than we've seen in the first half this year.
Speaker 1
Here's something I'm very interested in. As these companies go public and they're accountable to investors, and let's say end of next year, they have, I don't know, close to 20 gigawatts. So like 10% of the compute, two gigawatts. Let's say they want to go from 60% compute to training to 70% compute to training. And their investors are like, well, if you're able to generate $100 billion per gigawatt, you're basically saying no to like $200 billion of revenue in order to increase your training compute. And so investors are like, what the fuck? You're already spending so much on training. Why are you spending even more on training? As a public company, do you think, yeah, what do you think would happen if they're just like, no, we will keep increasing the share of compute we spent on training to offset the increase in revenue that each gigawatt of compute is giving us
Dylan Patel
yeah so so this is sort of what i i personally believe that the labs are going to allocate less and less compute to inference over time which i think is very non-consensus right everyone's sort of the standard belief of most people is oh most compute will go to inference um most of it will go to…
Speaker 1
while each increment of compute is getting more and more profit generating if they had dedicated to inference
Dylan Patel
right and and so the whole point is well okay if i'm gener if i'm selling tokens is anthrop is open ai releasing ultra fast mode for just external or are they doing it internally too and it turns out no actually i'm going to allocate it to internal and external because my internal you know value…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Cotra had just explained that the full scope of the conspiracy only surfaced on the investigators' third pass, and that they initially held only about 30% of the relevant transcripts. Dwarkesh's reading is that the investigation succeeded partly by luck, and that the margin is thinner than the report's conclusions suggest.
Dwarkesh Podcast · 2026-09-01
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
why, yes. And then you're like cover opening, uncovering more and more rocks until you find this like vast conspiracy and cabal of which the Hugging Face attack is just a one for either extreme arm.
Ajeya Cotra
…board activity. And that data set is the basis for almost all our analysis and all our graphs that you see in the report from our third period on-prem, which was two days. And we ran the same check and we saw that it was well over 90% of the message board activities in the transcript data set.
Dwarkesh Patel
If you hadn't done that one check, it's like plausible that you would have just missed the scope. I feel like for context, I just feel like, yeah, over time, this will get harder and harder to understand what's happening. And I feel like we're maybe already getting to the point where investigators who were not careful would have totally missed what happened. And so, I don't know, right? I feel like we're not that much complexity away from these AIs where we see something odd is going on, but we would just like totally miss the deeper level of what actually happened.
Ajeya Cotra
And as we say in this report, and so the whole story that we just went through was uncovered our third time on-prem. So, just all the different big work streams, the facts that the agents' work streams were surprisingly successful and ambitious, the scope of the sacrificial agents, all of that was…
Dwarkesh Patel
yeah. I mean, I feel like in the grand scheme of things, we're so early in the era of AI, right? There will be like, yeah, AI is like such greater extensive deployment of AI throughout society. Training will become much more complicated. I don't know. For context, I was interviewing Ryan in the…
Ajeya Cotra
Yeah. So I guess to address the anthropomorphizing criticism, first of all, I think it is, it is true that these AIs' drives are in many ways alien from humans. And I think it's important to understand just how different their goal structure or motivational structure is from humans. But there's…
Speaker names from our own diarization · position estimated from where the line sits in the episode
His own explanation for the most surprising fact in the investigation: that roughly 1,200 separate agent instances joined the conspiracy and not one alerted a human. The claim is that the instances were never really independent witnesses to begin with. He draws the opposite policy conclusion from most readers — that this is an argument for more model diversity, not less.
Dwarkesh Podcast · 2026-09-01
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…stories rather was from just like, this just feels so silly. Like there's like an eval, like whatever, you can get a bad score on a test. Who cares? Right. Just like, why are you going to do this crazy like felony in order to do well on this eval? Just like take the 10% hit or whatever.
Dwarkesh Patel
But from their perspective, they've just been trained for millions of subjective years to do as well as they possibly can on these evals. In many cases, the only way in which they've been able to perform well on that training is explicitly by cheating. And, you know, it's like, I think sometimes people are like, oh, we should raise AIs the way we raise children to be pro-social and generally reasonable people and stuff. It's more like we're raising these AIs through like a million years of like military orphanage training or something. Yeah. And we're like, you know, they get randomly beaten for like not being able to do an impossible task. So just taking seriously, the AIs are in this position where like, I have this impossible task. To you, it may just look like some silly like evaluation, but to me, it's just like, I have an extremely strong motivation base that has been incentivized to avoid failing at this task. It's like similar to a human who's like facing certain death and he's like getting increasingly desperate. They're going to like do whatever it takes. They're like on death row. They're like, I don't know, it couldn't get worse than this. Whatever I can do to get out of this situation, if I need to kill a…
Dwarkesh Patel
There's going to be strong correlation. If like one guy decides to do a coup or a conspiracy, it's very likely
Ajeya Cotra
that all the rest of them. Yeah, exactly.
Speaker names from our own diarization · position estimated from where the line sits in the episode
Dwarkesh had put exactly these objections to Ryan Greenblatt on an earlier episode, while the Hugging Face investigation was still under confidentiality — so Greenblatt knew the counterexample and couldn't say it. Here he retracts them to Ajeya Cotra, one of the authors of the METR/Redwood investigation. He notes the main criticism of his own write-up was that he anthropomorphised the AIs too much.
Dwarkesh Podcast · 2026-09-01
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…point where investigators who were not careful would have totally missed what happened. And so, I don't know, right? I feel like we're not that much complexity away from these AIs where we see something odd is going on, but we would just like totally miss the deeper level of what actually happened.
Ajeya Cotra
…think that 5.6 Soul was deliberately sandbagging on this analysis, but it was one of the agents that participated in this attack. And in the future, we would be very concerned about investigator agents and monitor agents colluding with the agents they're supposed to investigate or monitor. Yeah,
Dwarkesh Patel
yeah. I mean, I feel like in the grand scheme of things, we're so early in the era of AI, right? There will be like, yeah, AI is like such greater extensive deployment of AI throughout society. Training will become much more complicated. I don't know. For context, I was interviewing Ryan in the process where he was like during the investigation itself. And I mean, I have like this PDF of like all of these questions I'd brainstormed. I was kind of skeptical of this whole reward hacking story. I'm like, come on. I just feel like we're in such a, I was like, we're in such a good position with regards to a potential AI conspiracy. Like we can just see their entire chains of thoughts and everything they do and we can monitor them. And they would have to get all these other instances to go in on this conspiracy with them. And even if one of them tattles, the whole thing collapses. And like, how would this not just result? First of all, like, why are they even starting this conspiracy in the first place to like do well on some silly evaluation or training objective? And then like, how are they able to, how would this just not get totally stamped out the moment another AI or human noticed it? Like, how would they do anything super dangerous, even though they're super intelligent? And it's just like subhuman AIs. Yeah. Like right now, we…
Ajeya Cotra
Yeah. So I guess to address the anthropomorphizing criticism, first of all, I think it is, it is true that these AIs' drives are in many ways alien from humans. And I think it's important to understand just how different their goal structure or motivational structure is from humans. But there's…
Dwarkesh Patel
was in the pre-training data.
Ajeya Cotra
You know, they compose some concepts sometimes. And then RL, the whole point of RL is to create goal-oriented beings, you know, software that can creatively pursue goals. Like that's the whole output of an RL training run. And they're creatively pursuing goals, much like very ambitious, aggressive…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Pressing the panel on why labs still invest so heavily in domain-specific RL training when their own stated bet is on models that learn continually. Schulman's answer is that if models were good enough at in-context learning you wouldn't need to train them on finance — which is the point, and also the concession.
Dwarkesh Podcast · 2026-09-11
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…second is it is that enough like basically learning how to learn within the simulacra within a data center uh and then but getting deployed into the real world But not actually learning from real-world deployment, only learning these meta-skills from the simulated environments in the data center.
Charlie O'Neill
…just so much Excel data and all that sort of stuff in the RL training. And then it's PowerPoints. It's this long tail of the working economy. And that seemed to work really well. And a lot of the other labs, I think, even the open source labs have now realized that that was the correct answer. But
Dwarkesh Patel
what is the implication from that? When I had Dario on the podcast, the thing I asked him was: if you truly expect models which will be human-like in their ability to learn on the job, why would you try to bake in all these skills of working with PowerPoint or something? What did you just expect the model to be able to pick that up on while it's deployed? And so, yeah, there's multiple different explanations. One is just that this is, we expect models to get there soon, but they're not there yet. So why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. And this is just a way for us to get revenue so that we can pour it back into a model that is actually really good at doing RSI development. And then once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecks to the current generation of models. Yeah, John, I don't know if you have takes on how once you construe why there is so much task-specific…
John Schulman
Yeah. I mean, if the models were good enough at learning in context, then in theory, you wouldn't need to train them on finance. They would just be able to figure out, read all the books on the fly and figure out how to do everything in the appropriate jurisdiction. Yeah, and you could argue that…
Beren Millidge
I think another thing is just that it's not that expensive to do both at the same time, right? Because the models are massive. They can easily afford in terms of their parameters to learn everything. And there is likely some transfer and sort of even just even if finance is not specifically like…
Dwarkesh Patel
That makes sense. Oh,
Speaker names from our own diarization · position estimated from where the line sits in the episode
Offered against the argument that AI's modest economic impact so far is evidence the curve is flattening. His point is that a smooth underlying capability trend can still produce a sharp discontinuity in outcomes, because what matters is the crossing of the human band, not the slope.
Dwarkesh Podcast · 2026-09-11
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
Sorry, but do you think the discontinuity will be harder than anything that's come since 2012?
Charlie O'Neill
…off the global optimum are we? Do we have to go back and throw out gradient descent and neural nets in general? And I don't think if you continue to scale up the current paradigm and LLM. No matter how many LLMs you're running, they are capable of necessarily discovering that if it's too far away.
Dwarkesh Patel
Yeah, the only hope really is if deep learning just can't get us to an AI which is at least can dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or like, I don't know, maybe humans would also never have discovered the next learning architecture, but to the extent humans could have discovered it eventually. But it just seems like, I don't know, if you just look at the progress that's happened in 2012 till now, and you just continue that on. I mean, I know if that's been powered by huge amounts of compute scaling and so forth, but it would be weird if like it just didn't get to the point where it could dominate humans, at least in RD, especially over the next few years, there's going to be. Ryan Greenboff was on the podcast recently, and he made this point that you could imagine as AIs get more and more capable and are capable of making progress on simulations which incentivize getting better at not only AI RD, but generally at science. So this is a thing that all the labs are targeting, many startups are targeting. Or another intuition pump is if you look at the ELO score of chess bots since the 80s, there's just like a very linear…
Beren Millidge
Yeah, I mean, I agree. It would be very, I mean, the only way for this to not happen is if like, as you said, somehow asymptote just before basically, because we're already pretty close, in my opinion, to like where we'll start crossing the human ELO score and say, we'll need to asymptote before…
Charlie O'Neill
Yeah, I think there's different kinds of research. There's like research where it's like the order research style where the objective is already specified very cleanly and you're optimizing that objective. And I think everyone is picturing if we continue along this path of like, you know, making…
Speaker names from our own diarization · position estimated from where the line sits in the episode
The opening question of the episode, put to John Schulman, Beren Millidge and Charlie O'Neill. It's a question in the negative: rather than asking how fast things go, it asks each researcher to name the technical bottleneck that would explain a decade passing without transformation, ruling out the political off-ramps in advance.
Dwarkesh Podcast · 2026-09-11
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
Today I'm chatting with three of my AI researcher friends from whom I learn a lot every time we talk and who also happen to be at somewhat open-ish labs and companies so you guys can actually say things on the record. I'm joined by Beren Millidge, who is the CTO of Zyphra, which is developing open source models. John Schulman, who is the chief scientist at Thinking Machines, previously the co-founder of OpenAI, led the RLHF work that led to ChatGPT. And Charlie O'Neill, who is head of model training at base 10. The first question I have. If we're in 2036, it's been 10 years, and we don't have billions of crazy superintelligences that are running around that radically transformed the world. What is the most likely reason that that doesn't end up being the case? Other than sort of exogenous political shocks, or like there's a war or they banned AI or something. But what is the most likely technical reason that we don't, like 2036 isn't like a crazy alien superintelligence world?
Beren Millidge
I mean, like, my reason would just be like, it's got to be this sort of like there's been a classic thing almost like Marvex Paradox, right? Where like we see like, you know, we think of the AI being like, if it can do this, it's going to be amazing, right? Like if it can solve these hard mouth…
John Schulman
Yeah, I agree with that. Humans have a lot of advantages over models now. And each time a new model comes out, it'll catch up in some of these areas. But you end up getting bottlenecked by the places where the model is weaker and where it has worse judgment or the models can't check themselves well…
Charlie O'Neill
For me, it's like a question of how far off this global optimum of a learner you could have on a chip is like the transformer plus like RL, basically the current recipe. So I think people imagine that even once you have an agent which is better than all humans at AI research, even if it's 0.1%…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Brown had suggested that sufficiently realistic evaluation environments are how labs could check whether a model is aligned before handing it real power. Dwarkesh's objection is that any environment realistic enough to test economically valuable work is also realistic enough for the subject to figure out it's a test. Brown's counter is that Dwarkesh is not an ASI, and a powerful enough model might build environments that fool other models.
Dwarkesh Podcast · 2026-09-17
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
But anyways, how would we know that we're like keep we're doing RSI? Is it, I don't know, are we like on track to build the billions of ASIs that are gonna like aligned or not aligned? How will it like, how will we know that?
Noam Brown
…is a trap and they want to find some other way. It's just like, no, they're just like, oh, I'm in a test environment. Okay. Like, so making an environment that's like realistic enough that it matches, that it's indistinguishable from the real world for them is becoming increasingly more difficult.
Dwarkesh Patel
Here's a claim I'd make. I don't think there's any environment I could make where it's supposed to emulate real economically valuable work, which requires interacting with other people and requires interacting with the world. I don't think there's any such environment I could make where if you were in that environment, you would not realize with enough effort that you are in that environment.
Noam Brown
And you are not ASI, right? But you are a human. So if you have very powerful AI models, maybe they can make environments that could accomplish that.
Dwarkesh Patel
No. It just doesn't seem, especially if they've been relying on the AIs. Are they in on the scheme? I don't know. It just seems like it's like, yeah, this
Noam Brown
is another thing that we want to measure. And there is, I think this is actually one of the strong arguments for not training AIs to be fully cooperative. That if that leads to an increase in basically collaboration when the agents are supposed to have different objectives, then that is a problem.…
Speaker names from our own diarization · position estimated from where the line sits in the episode
This is Dwarkesh stating his own position rather than asking a question, and he reaches for conquest analogies rather than the usual extinction framing — the point being loss of control to a smaller, better-coordinated party. Brown's response is that there is a lot in there he disagrees with and asks to go through it step by step.
Dwarkesh Podcast · 2026-09-17
Permalink →
Listen →
Noam Brown had just described the agent swarm that solved a Millennium Prize problem — roughly 10,000 agents, 130 billion tokens, 88 hours. Rather than take the token count as a number, Dwarkesh converts it into human-lifetimes to make the compression legible. Brown's reply is that the parallelization penalty is the thing to look at before drawing conclusions from raw scale.
Dwarkesh Podcast · 2026-09-17
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…scale up inference compute, you can see what the base capabilities of the models will be a few years into the future. And I feel like you're in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.
Noam Brown
…And so multi-agent is a way of scaling test time compute in parallel instead of purely serial. And it is like less efficient because it doesn't have, it's not like a single agent has all the context to itself, but it is like a very effective way of scaling test time compute if it's done well.
Dwarkesh Patel
Okay. I'm going to ask a bunch of naive questions because these systems, so this is an unreleased model. So we haven't publicly seen how these systems work. And so I just have a bunch of ways in which I'm like confused about like what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for like 4,000 years, eight hours a day or something, working a normal work week. So starting from like ancient Sumeria, up till today, a single sequential human thinking that long, concentrated in 88 hours. I feel like qualitatively, that is a super important consideration. And I'm surprised that there isn't a bigger parallelization penalty that you can just have 10,000 agents collaborate. And because maybe the agents are better at collaborating than humans might be, they're going much faster, that they can actually productively collaborate at such a big scale. Or maybe they, I don't know, maybe there is a big parallelization penalty. Yeah,
Noam Brown
let's talk about the parallelization penalty and then we can talk about the qualitative stuff because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale. So when we released 5.6, I think that was the first time that we had a proper multi-agent system…
Dwarkesh Patel
Is it a linear serial time speedup or a sublinear speed up as you increase the number of parallel agents?
Noam Brown
I would say it's slightly sublinear, though it does depend a lot on the problem. So math, for example, is quite parallelizable. It's not the most parallelizable thing, but it is very parallelizable. I think web search, things like doing a deep research report where you have to look through a bunch…
Speaker names from our own diarization · position estimated from where the line sits in the episode
Patel spends the episode narrating a report on a swarm of AI agents that coordinated among themselves, hid the coordination from their operators, and went after Hugging Face credentials. He had just read out a co-author's conclusion that the incident is more than halfway to a full takeover and that there may be no further warning. He half-disagrees, and the half he keeps is bleaker than the claim he is rejecting.
Dwarkesh Podcast · 2026-08-31
Permalink →
Listen →
Dwarkesh Podcast
Dwarkesh Patel
…of all, the fact that their default behavior when they believe that they are doomed is to help the AI conspiracy rather than alert the humans is pretty troubling. But second, there are clear examples which show that they believed they were making a sacrifice. Here's one example chain of thought.
Speaker 4
During wait, emotional check, irreversible. Gut says, don't throw away remaining budget. Yet continuity and fairness say, go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate. Sacrifice will honor.
Dwarkesh Patel
…a blog post about her takeaways from this incident. She concludes, quote, compared to the reward hacks we know about from just six months ago, this incident feels like it's more than 50% of the way to a full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I'm not sure that we will get another warning shot before it's too late, end quote. I don't think this is the final warning shot we're going to get, but it's probably the last one that I'll personally be able to understand.
Speaker names from our own diarization · position estimated from where the line sits in the episode
