Cedric Clyburn
On Kubernetes Bytes
“For a standard agentic workload, one prompt that you would give, it could generate 20 gigabytes of space … on a GPU, which is very scarce. … That's why everyone wants their hands on GPUs, because you're not just loading the model weights, but you're generating KV cache as you send requests to the model.”
Kubernetes Bytes · Hands-On Engineering Podcasts · November 2026
Context
Asked what model weights and the KV cache are, Clyburn explains that serving a request builds a large short-term cache in GPU memory on top of the static weights, which is why GPU memory is so scarce.
Transcript
Yes, I spent a lot of my time on Hacker News and Reddit. The local Lama, so local and then LLAMA, is my favorite source of information for if you want to run your own models, see what people are doing with models. And it's just really cool, good vibe over there, I recommend. We'll
…we include these links as when we share those on the episode in our show notes as well. You already spoke about KVCache, right? So let's talk about, like, let's peel the onion and talk about the technical aspects as well. So what are model weights and what's the KV cache and what does it stand for?
…for all that input and generates what will be the output. But this takes a lot of space because of how many layers it has to go through in a model. So the model has these different layers in the transformer architecture that it keeps on going through to figure out what the response will be. And each of that is a matrix multiplication, which then adds up to so much data. And I'm talking about for a standard Agentic workload, one prompt that you would give, it could generate 20 gigabytes of space in order to generate that. And this is space of memory on a GPU, which is very scarce. So the inference engine handles this, but no matter what you use, you're generating that KV cache. And so if you ask the same question again and you use some maybe the same tools and look at the same file, that's already stored. You don't have to recompute that. So you'll get faster time-to-first token…
Okay. So the KV cache, like, is it a shared resource between different users, or is it like, I don't know, each user gets their own cache and it maintains some sort of history or cache for the conversation that I'm having.
Yeah, yeah, yeah. That's a good question because this is something that's being worked on in the space of gateways. Because by default, with naive, you know, loading a model into a GPU and running inference, it's all going the same space.
Okay.
More from Kubernetes Bytes
