“I don't think that move 37 was creative. I think that move 37 was innovative without being creative.” Edward Hughes · Machine Learning Street Talk
AI Research & Frontier Labs · September 2026
Tim Scarfe had just said he assumed the two of them agreed that AlphaGo's move 37 was creative. Hughes contradicts him on the spot and spends the rest of the turn separating the two words: innovation is turning unknown unknowns into known knowns, while creativity requires someone to recognise afterwards that the thing done was creative. His follow-up question is who did that recognising for move 37.
Transcript
Yes, I think there's a lot of that I agree with. Maybe I'll unpack it backwards. So I think the one wrinkle I'd add to that at the cultural level is I don't believe that there will be some separate culture for agents and a separate culture of humans. In fact, I think the super intelligence, insofar as it will exist, will be a combination of humans and agents interacting in very deep and very complex ways. And that knowledge will be the better that we're able to interconnect agents and humans and leverage their complementarities, the faster we'll be able to drive this accumulation process of knowledge. Now, I think that absolutely that there is this distinction between, if you like, the slow weight updates, which are maybe more akin to the evolutionary process for DNA, and then the fast in context updates. I suppose one of the things I'm very interested in at an architectural level is whether this analogy will still hold in five years' time, for instance. Now, a lot of pieces, if you like, that could sit in between those two, epigenetic effects, for example, are one. And what's the analogy for epigenetic effects in an AI system? So, one of the pieces in our new paper that we talk about is this idea of a weaker coding agent using a stronger coding agent as a tool. And there, what we do is we actually update the weights of the weaker coding agent because that has the benefit of generalization more so than just updating the context. But because the weaker coding agent is using the stronger coding agent, that weaker coding agent is, of course, injecting things into the context of the stronger coding agent. So, now you have to ask yourself: well, is this weights or is this context? And the answer is, of course, it's both. And perhaps we've got the opportunity to have a much more intricate spectrum between these two things than we currently have. And that's one of the themes that we're in particular investigating because, in the cobbled together nature of our current AI stack, what happens is you get these abstractions that become very sticky, rightly so because they work well. But because of the burden of knowledge for humans to understand the frontier, we just accept a very large number of these abstractions because it's just too complicated for us to be examining all of them in combination. And the promise, I think, of AI scientist systems, agents that really understand this horizontal, is that they might be able to weaken multiple constraints at once and thereby enhance the ability for creativity.
Another loop that you opened earlier was: you know, you spoke about move 37, and I think you and I would agree that that was definitely creative. It feels like there are some limitations to its creativity. So, I would call it a form of concrete creativity. So, it doesn't understand in the sense that Margaret Bowdoin would speak about in terms of understanding how it hangs together in the context of the system and what is possible and counterfactuals and whatnot. But it's still a form of concrete understanding. But another interesting angle there as well is you were talking about the difference between possibly human knowledge and AI knowledge, because I was speaking with Tom McGraff at Goodfire. He did interpretability on AlphaZero. And his idea is very much that these things are learning the space of human concepts and beyond. And we could actually mine those representations as a new form of science. So these things are discovering interesting things that perhaps we would discover but haven't discovered yet. And we could actually use this as a laboratory for discovering interesting new knowledge.
Well, interestingly, I don't think that move 37 was creative. I think that move 37 was innovative without being creative. That I think about innovation is innovation is the process of taking unknown unknowns and making them into known knowns. And in order to get an innovation, you can't generate an innovation if you sort of already knew what it was you were looking for. You have to have something that's unexpected, but that also becomes valuable. So, what's the difference then between innovation and creativity? Well, in my mind, creativity requires another step, which is to recognize that the thing you have done is creative. And who was it who recognized that move 37 was a remarkable move? It wasn't the AlphaGo, it was the commentators, for example, who, if you watch the famous footage, say, oh, that must have been a clicko. Was it a click miss? It wasn't, of course, it was exactly the right move. And so, that is a kind of meta-metacognition. So, you have to sort of understand that you didn't know something, and now you are now updating your own knowledge as a function of that. And it's interestingly discussed by Mihali Chiksetmihali. He wrote a lovely book called The Psychology of Creativity and Invention. And he's also the person behind flow. So, many of your listeners will already know a concept from him. But in this book, he interviews a very large number of different creative people from different disciplines. And he comes up with an ontology of what creativity is. And he says creativity has got three components. There is the creative individual. That's the bit that we always focus on. But that's really in some sense the tip of the iceberg. The second piece is the domain. And the domain is a set of symbolic rules, if you like, to which the creative person is adding or perhaps breaking one of the rules and then thereby expanding the space of possibility. But the third piece, which is perhaps the most forgotten one, is the field. And the field are the set of other individuals who are going to decide whether the creative person's contribution gets admitted into the domain. And he gives this lovely example of Florence in the Renaissance. So we're talking 15th century. And in the 15th century in Florence, there was an enormous flowering of creativity, whether it was architecture, science, even the way that society itself was structured. And the question is, what was it that led to that in Florence? Now, you might say perhaps it was just an expansion in the number of creative individuals. Was there some mutation of the DNA, some new educational system? It seems quite unlikely that there could be a mutation in the DNA. And so far as we know, there was no great change in education. Well, then you have to ask, okay, was it the domain? And I think at least in part it was the domain because at that time, many building techniques that had been lost to antiquity, which were in fact known to the Greeks and Romans, were being rediscovered via archaeological means, via analysis of the building structures that people were uncovering. But it couldn't just have been the domain because much of this rediscovery was happening in Rome. And Rome didn't have the same flowering of creativity as Florence. So the third thing you need is the field. And what Florence had, but the other Italian cities didn't, were lots of very rich families. The Medici is the most famous among them, but I didn't believe it was the richest. And they were rich from the wool trade and then also from becoming financiers as well. And they had this idea of making Florence the most beautiful and most cultured city. And that was in some sense to weave a protective cloak around the city at a time when there were many city-states, there was quite a lot of conflict. And they believed in this idea of beauty as in some sense a kind of psychological defense. And as a result, there were a very large number of creative constructions that were admitted into the domain. And it became a competition between the artisans of the day. And so I think it's instructive to think about how that might play out with AI scientist systems. And in particular, I think it becomes much more interesting when these AI scientist systems start to be able to do things which are generalizable. So what do I mean by that? Well, of course, move 37, remarkable innovation, but it doesn't really tell you how to do innovation in other domains. We didn't immediately see a line from move 37 to discovering a new material, for example. And even if you think within their single organization, the line between move 37 and say alpha fold. Wasn't a particularly direct line. It's not like the AlphaGo agent or indeed the AlphaGo training techniques really informed AlphaFold in a very direct way. But in principle, if you had had a generalizable discovery engine, then it could make a discovery about the weather and then figure out, oh, there's some part of that discovery. Perhaps it's the architecture of the neural network that was used in order to make that model. I wonder whether that applies to protein design. And those kinds of connections, I think, generalizable connections are going to be what leads to a large acceleration in the rate at which we can make discoveries.
Yes, I mean, because you were discussing how we recognize creativity, and the social component is extremely vexed because it's very tempting to think there's some degree of social proof in creativity. And indeed, perhaps there is. I mean, there's a famous example of a Urinal with a bit of masking tape on, and everyone just decided that it was creative. And I tried to think about it abstractly. So for me, something is creative when it becomes a mode in the state space to a certain extent. So that clearly became a social mode. I mean, what's difficult about us as individuals and cultural learning is the introduction of agency and the fact that we could have done differently. But I suppose going all the way down to physical creativity, you know, evolution isn't an agent. It's not doing planning, but there are still these canalized modes. And the way I think about it is it's a bit like the system has discovered an interesting new subspace, and that subspace is being used in a myriad of situations. And so we would call that discovery creative. And perhaps even with AlphaZero, maybe if we enumerated many possible game, many possible game trajectories, and if we saw something that looked like a category, so this particular type of pattern was being rediscovered, reused in many different situations, we could immediately look at it and just draw a boundary around that category. But maybe AlphaZero would kind of have competence without comprehension. So if it was using this thing in many different situations, maybe then we would call it creative.
Yes, I think I want to come back to the idea of a relationship between creativity and constraints for a moment. So if we look right back at evolution itself, I think evolution quite clearly is creative. It's certainly generated this enormous amount of diversity in the natural world. And the way it's done so is exactly by satisficing. Satisficing is just a posh word for saying satisfying constraints. So why do I say it's satisficing rather than optimizing? Many people might think isn't evolution trying to optimize for the best, the best of an individual with the most adaptive traits. Well, actually, all that evolution requires is that individuals survive and reproduce. And once you've done that, there's not a lot else that you can do. Now, perhaps you could say you can do second order survival and reproduction. That is true. You probably care about your children surviving and reproducing as well. So it's not quite as simple as that. But even at that second order, that's still a constraint satisfaction problem. And there's two interesting implications of this. The first one is if you really believe that creativity is satisficing, then the default crutch that we reach to in machine learning, which is optimization, is the wrong thing to reach for to build creative agents. And secondly, if you believe that constraints satisficing is important for creativity, then the way that you open up new creative spaces is that you take your existing constraints and you break some of them. Now, why do I say break some of them? Clearly, if you break all the constraints, then there's no meaning left, right? You know, the way that we construct meaning is, and indeed the way that we construct laws of the universe, is that we rely on things being repeatable. This is the so-called principle of induction, which is not something that you can prove, but it's something that we just observe. The laws of the universe seem to stay the same from moment to moment. So you can't break every single constraint. Otherwise, we'd live in a world of white noise. But breaking some of the constraints is very useful because at least some of the constraints arise because of our existing theories. We don't have access to the universe. We only have access to the universe through our observations and measurements of it. And in order to make those observations and measurements, we do two things. We have tools that allow us to make those measurements. And then we have our own neural apparatus, which allows us to make interpretations of those. And so by relaxing or breaking some of those constraints about the interpretation or about the tools, we're then a Able to access new insights about the universe. And so that's where creativity really arises.
Yes, you've opened so many loops there. I don't know how to close them off in order, but I'll try my best. The thing that you just said is very interesting, which is this very vexed issue of coherence, right? I mean, atonal harmony is the great example of this. And I often argue with my co-author on this article that we wrote, whether that is breaking the constraints or inverting them or just respecting them in some other way. Because a lot of people talk about knowledge being quite situated. And what they're meaning in that case is that it's only coherent if you respect the constraints. And sometimes it's not possible to break the constraints. And another thing you spoke about, and this is also related to your 2024 ICML paper, which was open-endedness is, I think, necessary or required for SS. Yes. A beautiful paper, by the way. And I said to Tim Rock Tashel at the time that I felt that was actually a definition of creativity rather than open-endedness. Actually, I think they're basically the same thing. And the reason I think they're the same thing is that intelligence is basically about optimization, right? So intelligence is like, you know, I'm trying to find the shape of the maze and I don't know its full shape yet. And I'm trying to fill it in and I can go in that direction. Creativity, as you were saying before, it's about discovering new questions, new problems, new mazes. And as Kenneth said in his book, Why Greatness Cannot Be Planned, there's a weird paradox there that when you optimize towards something, it's really, really difficult for you to find something interesting and creative because you've got the blinkers on.
More from Machine Learning Street Talk
