“I tend to want to use the most expensive model available always because, like most capitalists, I equate trust with cost.” Ken Mugrage · Thoughtworks Technology Podcast

Using the food analogy, for example, OPEX versus Scapex, right? So, OPEX, it will be the equivalent of ordering food delivery. So, you are kind of renting access to closed models like OpenAI and Tropic Google via APIs. So, you pay like a tiny fraction of a cent per input per token and output per input and per token, essentially. The advantage is like zero upfront cost. So, you don't need to buy expensive computers, hardware, or have a farm of Mac Minis in your company or pay massive electricity bills. You just pay for what you consume. So, pay-per-use, basically, right? But with a caveat, as your application grows from hundreds of users to millions of users, your monthly bill grows linearly. So, a little spike in traffic, this translates directly to a massive recurring monthly cloud bill, you know. So, that would be costly for you. But on the other hand, if you want to use an open weight model, it's the equivalent of, okay, I'm not kind of wanting to order food anymore. So, I'm going to build my own commercial kitchen, so to speak. So, you make a large potentially one-time investment in local physical hardware. So, this includes, you know, buying desktop, supercomputers, like NVIDIA, GX Spark, or Mac, Mac Studio, Mac Minis workstations with unified memories, or even a full enterprise GPU server racks, right? The advantage on that is that once this hardware is paid for, running inference on an open weight model like DeepSeq or GLM, this costs virtually nothing beyond basic electricity. So, your cost per token drops to near zero, you know. So, and that also other technical advantages like the latency is lower, you know, because there will be no round trips on network or third-party servers. Sensitive data stays entirely inside your building on your machine. So, yeah.

The other reason people use Frontier models, though, is that let's say Anthropic and OpenAI are some of the most familiar Frontier models that we associate with them today. They have incredible funding and incredible researchers and an incredible head start. And so, those models tend to be more performant and more capable than the OpenWait models, which have generally smaller funding, smaller teams, sometimes unpaid teams, that kind of thing. And so, for the long time, Frontier model has directly equated to the best model. But as we'll probably talk about over the past year with DeepSeek and the GLM series and the Quinn series, all these models are starting to catch up rapidly to Frontier. And so, now the conversation is shifting away from good model versus bad model and more towards model that somebody runs for you and model that you can run yourself.

I will say I am guilty of only using Fable when it's available to me and not using Opus or Sonnet or Haiku. But what I've noticed recently, over the past few weeks, I started to play with the GLM 5.2 models, which 5.3 has made waves for being almost as good as Fable, if not as good, to the point where there's some controversy about did they try to distill things from Anthropic and learn how they did it from the outside, which is a little, that's a little sketchy if they did that. But either way, that's an open weight model that competes performance and performance with Fable, potentially. And so, what I'm noticing in my own work is that the open weight models are getting to a point where her most general tasks, like searching knowledge, summarizing documents, or writing very well-scoped and well-defined code code changes, the open weight models are just as good, at least as Opus, maybe as good as Fable in some cases.

What does it take to run some of these? I mean, like, I early on tried to run some of it on my MacBook, and I have a pretty decent MacBook, you know, 48 gigs of RAM in the big processor, the big GPU. Um, and most of the larger stuff, I couldn't even start. I mean, unless I didn't know what I was doing, which is equally as possible. But what's it take to really run these?

I will say. These models, the ones that are truly competitive with frontier models like Opus and Fable, we're going to have a hard time running them on a private computer. If you have a specific kind of Mac Pro that has, say, 500 gigs bytes of VRAM available to it, which they exist, but they're unavailable and unobtainable now for obvious reasons. If you have one of those, you could run these models locally. If you have a farm of Mac Minis, like Andre was saying, you could potentially run one of these with distributed inference, but it'll be a lot slower because it's having to do consensus and all the fun problems that come with a distributed system just to get you tokens. So, what I do, and probably a lot of people do, is you can run openweight models off of other cloud providers. So, you're still paying for your tokens, you're still paying an amount of money to do inference, but you are paying a lot less upfront than having to buy a GPU, for example. And generally, you're paying a lot less that you pay when you're using a Frontier Labs models. Like, as an example, these aren't the actual numbers, but Fable feels like it would be like $20 per million tokens, whereas an OpenWait model might be $1 by comparison. The actual numbers, who knows? But it's that kind of difference that you see.