“The overlap of problems that are easy enough that they are suitable for someone who is learning to do, and hard enough that an LLM can't just one-shot it, is close to the empty set now.”
Signals and Threads · Hands-On Engineering Podcasts · September 2026
Minsky and Aaron Bauer were discussing how Jane Street teaches early-career engineers now that an LLM can write the scaffolding for any exercise they set. Minsky's worry is that struggling with the material is a core part of learning, and that the exercises which are both small enough to teach with and hard enough to resist a model have nearly vanished. His tentative answer is that education sometimes works by withholding affordances — like making people do it on paper.
Transcript
The initial scaffolding can be made really easy, right? The models can do a decent job of that. But what's the understanding that you have to give to people?
Yes. And so I think so far we've come down on the side of people having to think through these sorts of problems and implement them themselves at least once is actually pretty important for coming out of it with a sufficient understanding. And even if what they are doing subsequent to that is reading applications of this technique initially authored by an LLM, building that understanding by having done it themselves once is sort of a critical input to then being able to evaluate those things. That is one piece of it. And I think we are actively thinking about where to draw the line in terms of there's some stuff that we were probably asking people to do in the past because it was sort of stuff that they had to do, even though it was somewhat toilsome. That, yeah, we should probably not have kind of people try and learn how to do those pieces. But there's also, I think, the challenging question of how do we teach people and even what do we teach people about using these tools effectively, about how to integrate them with their workflow, about how to understand the ways in which they can be effective or lead you astray. And I think this is still in very early days. I don't think we have sort of settled answers to this. And the tools are changing very quickly.
But you can already see problems, right? There are definitely people who run into problems where they just like kind of vibe it up too much and like use the LLMs to generate stuff and do not think. And like that's a real problem. Like you still need to think. I actually worry more generally about what it means for the overall education system. Like I think a kind of critical component of learning is struggling with the material. And the world where you always have a genie at your beck and call that you can just like summon up to solve the problem is itself like a real problem. And the overlap of problems that are easy enough that they are suitable for someone who is learning to do and hard enough that an LLM can't just one-shot it is like close to the empty set now. So it's like a really challenging problem. I mean, it really does make me just think back to the like doing stuff on paper without a computer. Sometimes an important thing about making the educational process work is withholding certain affordances. And I worry that we have like a whole generation of people who are like learning in this new environment where we don't know what to do. We don't know what affordances to provide and which ones to withhold. And like I worry that we're like not building a good educational system for people in this world.
Yeah, I think you and pretty much everyone else involved in education is deeply worried about this. To your first point about like, how do you get something that's small enough to be tractable for students and large enough to not be trivial with an LLM? Like we have run into this exact problem multiple times with new curriculum we've built and we've sort of repeatedly ended up the thing we're asking people to do is too large for the time we have. Like we are asking them to do something where they do have to put a lot of work into the LM code, LLM code to make it good and actually do what they want. But that just takes too much time to do well, given sort of the amount of time we have on the education piece. And I think it's going to take more experimentation to figure out where that lands. But yes, I think that figuring out where to slot using these tools into sort of traditional education of it feels inevitable that we're going to end up with some sort of structure where you do this piece or this set of activities without the aid of LLMs and then you do some with. And I think figuring out kind of how to make that pairing and what you're trying to get out of each of them is something that a lot of people are going to be experimenting and figuring out.
Yeah, and I think it'll be interesting over the next few years as we're doing this increasingly with cohorts of people who have not spent that much time. Much time typing code themselves. They'll just be like, there's like a core learning mechanism that maybe they haven't had enough experience with. And yeah, I wonder if that will change what we have to teach people in the end.
Yeah, there's been a lot of conversation about things along the lines of that cloud code learning mode, where what LLM tools, if any, should we provide to folks going through OCaml boot camp, learning OCaml for the first time. And if they're just having LMs write all the code, well, it doesn't seem like they're going to learn anything. But it also feels a bit odd of people going to leave this and then LMs are going to be a core part of their day-to-day work. And so not having those be at all a part of this also feels like maybe not a great match for what we want to teach them. And so can we come up with prompts, something, a mode where the LLM will just sort of ask you questions to check your understanding, or we'll leave you large pieces to do yourself. Like we'll sort of set up a kind of skeleton, but actually ask you to do most of the implementation. And I think it's going to be a lot of experimentation with that as well.