Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.

Start free trial →
DS Danila Shtan On Unsupervised Learning with Jacob Effron

“As an industry we didn't optimize shit, pardon my French. I strongly believe that running inference, particularly with the variety of open-weight models that are available today, is not a solved problem. If you take something and just run with it as it's published, your results will probably be underwhelming. It's not that fast, it's not that cheap.”

Unsupervised Learning with Jacob Effron · AI Builders · November 2026

“As an industry we didn't optimize shit, pardon my French. I strongly believe that running inference, particularly with the variety of open-weight models that are available today, is not a solved problem. If you take something and just run with it as it's published, your results will probably be underwhelming. It's not that fast, it's not that cheap.” — Danila Shtan, Unsupervised Learning with Jacob Effron

Context

Jacob Effron asked what low-hanging fruit remains in inference. The Nebius CTO argues open-model serving is far from optimized and that kernel work, pruning and compression leave a lot of room.

Transcript

Unsupervised Learning with Jacob Effron Around 24:05 into the episode
Danila Shtan

…a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like intentional disbalance of pre-fill and decode nodes? And can you fit more than one into a single rack? And how do you solve the packing problem?

Jacob Effron

Yeah, no, it makes sense. mentioned there's like lower hanging fruit on the uh on the inference side you know that than doing that maybe what what are like some of the lower hanging fruits today that you still think we're we're not uh as as as an industry optimized on

Danila Shtan

well first of all I think as an industry we didn't optimize shit so like pardon my French but in a sense I strongly believe that running inference particularly in the like variety of open weight open source ish models that are available today is not a solved problem like if you take something just run with it as it's published like in the initial support in some of the open source inference servers run with the novel models your results will probably be underwhelming it's not that fast it's not that cheap especially with like current prices on the infrastructure it's easier to go and use like i don't know fable 5.1 astra 6 etc etc etc but because of the open architecture because like of everything that we know as of today you can optimize the way you process those requests you can optimize the way the inference server is configured you put a lot of effort in kernel optimizations sometimes kernel over optimizations based like on the particular pattern that you see in the traffic right you there is a lot of underexplored potential in model weights optimization and i'm not just talking about like stuff like speculative decoding for…

Jacob Effron

there's a ton of work around inference optimization and a lot of that gets done in open source right or like you know contributions that that share throughout the community there's obviously a bunch of work on on like smaller purpose-built models and and compaction and a bunch of these things…

Danila Shtan

we want to be at the forefront of everything in a sense these are additive and even maybe multiplicative right like if you have a right subset of things optimizations that you introduce for a particular use case it means that you can reach the targets easier better faster in a more predictable…

Jacob Effron

I guess, like, you know, maybe along those lines, you guys have done a few acquisitions, and one of them that caught a few people's eyes was Tavili in the kind of like web search space. Maybe talk through a little bit the rationale for that, how you think about that fitting into the broader…

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Unsupervised Learning with Jacob Effron