Masad and Alex Atallah were talking about why enterprises may come to prefer small, specialized models whose outputs can be tightly controlled. Atallah had just said that structured-output models leave much less room for misbehavior. Masad's reply is a programmer's nostalgia for software that did only what it was told.
Would they be willing to pay 10X to get that much?
I mean, it probably depends on the types of tasks they're trying to do. Like, you know, some just have way lower risk than others. Writing code that or doing like security research is the highest risk type of task today. And so you probably spend 10X to get a fully aligned model that can also find all the bugs or fully like anti-deceptive model that can also find all the bugs. One of the coolest things about decision models is that you fully control the structured output. And generally with structured output models in general, like the room for misbehavior is so much lower. You just have like defined tasks and only machines are like dealing with the outputs. And it's not writing code that can execute. Those tasks feel like probably underrepresented in the ones that people talk about and in the things that enterprises are dealing with. So I expect enterprises to get a lot more interested in them.
Yeah, I feel like we're going to slowly realize how good we've had it with deterministic code. Like, oh my God, remember the days when computers did exactly what we told them to do. And I think things like Jev, I think, hint at more of a need for not only specialized models, but models whose output domain is more controllable. And maybe you could do a lot more than we thought you'd need by using a bunch of specialized models, specialized output models.
Have you guys done any work? Workloads internally with it.
You know, I've been training a lot of small models. I mean, I said this glib thing when it first came out because I was like, sort of, I gave this Hacker News comment-like comment, which I felt disgusted with myself afterwards. But I've been taking a lot of like Quinn 8B and like asking, honestly, like asking Fable and Opus and Astra to train a model. For example, I trained a cost estimator model internally so that when you put it in a prompt in Replit, we know exactly how much it will cost. And it basically emits a probability distribution over multiple buckets. Like if it is between $5 and $10 bucket A, bucket B between $10 and $20. So I'm used to training these classifiers by giving it different enums essentially and looking at the log props per enum. I've been doing it for a couple of years. I trained a chat spot to play by just doing that. So I'm already sort of pilled on like sort of decision models and specialized models. So it wasn't that big moment for me. But I understand that a true foundation model that's fully promptable is like, is like amazing user experience, amazing developer experience. And you can do a bunch of stuff with it without training a model from scratch. But if you have a data, if you work at a place where you have the data, you have so much data at Replit, I ended up training a lot of specialized classification models pretty easily.
Yeah. Like I definitely, it also feels like less model debt. Like something that I still hear from companies is that they're like worried about fine-tuning models for like unstructured outputs because you're just like always, you got to redo it again in like two months and everybody just feels the model, the weight of the model debt. But like a very bespoke classifier that's trained with like proprietary data, you just, I feel like people won't think it's behind constantly and it might just like last longer.