Asked how a stateless platform like Kubernetes can serve what feels like a stateful chat, Vacchi explains that the model resends the whole history each turn, and prefix and KV caches are what avoid recomputing it. That is also why cached tokens are cheaper.
Yeah, Anthony, LLD allows you to make cloud-native inference and evolve your infrastructure in such a way that you can do AI on premises in a similar way as you did cloud-native development before AI.
…that theoretically should be stateful? Because if I have a conversation, I would like to continue my conversation about, I don't know, whatever stupid thing I speak to Gemini or Cloud or OpenGPT or whatever stuff. I don't want to have my responses coming for what Eduardo was speaking to his LLM.
…discovered if you wrote that prompt as a conversation, as a dialogue with, you know, with quotes, user says, da, da, da, computer says, the LLM will continue the conversation. And that's how ChatGPT basically was born. Like people noticed this and noticed this pattern. Now, LLMs are being trained with this pattern, but that's how basically originally it worked. So if you think of it this way, every conversation really is a very long prompt to the continuation of that conversation. The way LLM works, you send the history of the conversation and then they continue with the next answer. And then you get back the answer and then you continue with your answer and so on, so forth, so on so forth. If you are using LLMs, you will know that pricing for these things, depending if you're using token-based spend or a subscription, well, if you're using a subscription, that doesn't…
Okay. So LLND, it's also donated, if I remember correctly, to the CNCF Foundation, right?
Yeah. It recently reached 1.0 a few days ago. Yeah.
Okay. Congratulations. So we are right in time. In what state is it? Is it production ready? Should we wait for it?