Connect your Claude, Codex or OpenClaw for full transcripts on millions of podcasts. Two week free trial, cancel anytime.
Start free trial →Unsupervised Learning with Jacob Effron · AI Builders · November 2026
Jacob Effron asked what low-hanging fruit remains in inference. The Nebius CTO argues open-model serving is far from optimized and that kernel work, pruning and compression leave a lot of room.
…a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like intentional disbalance of pre-fill and decode nodes? And can you fit more than one into a single rack? And how do you solve the packing problem?
Yeah, no, it makes sense. mentioned there's like lower hanging fruit on the uh on the inference side you know that than doing that maybe what what are like some of the lower hanging fruits today that you still think we're we're not uh as as as an industry optimized on
well first of all I think as an industry we didn't optimize shit so like pardon my French but in a sense I strongly believe that running inference particularly in the like variety of open weight open source ish models that are available today is not a solved problem like if you take something just run with it as it's published like in the initial support in some of the open source inference servers run with the novel models your results will probably be underwhelming it's not that fast it's not that cheap especially with like current prices on the infrastructure it's easier to go and use like i don't know fable 5.1 astra 6 etc etc etc but because of the open architecture because like of everything that we know as of today you can optimize the way you process those requests you can optimize the way the inference server is configured you put a lot of effort in kernel optimizations sometimes kernel over optimizations based like on the particular pattern that you see in the traffic right you there is a lot of underexplored potential in model weights optimization and i'm not just talking about like stuff like speculative decoding for…
there's a ton of work around inference optimization and a lot of that gets done in open source right or like you know contributions that that share throughout the community there's obviously a bunch of work on on like smaller purpose-built models and and compaction and a bunch of these things…
we want to be at the forefront of everything in a sense these are additive and even maybe multiplicative right like if you have a right subset of things optimizations that you introduce for a particular use case it means that you can reach the targets easier better faster in a more predictable…
I guess, like, you know, maybe along those lines, you guys have done a few acquisitions, and one of them that caught a few people's eyes was Tavili in the kind of like web search space. Maybe talk through a little bit the rationale for that, how you think about that fitting into the broader…