“You will find every other engine out there will employ weird mathematical equations to skip a few bits here, skip a few bits there, and it makes it quicker. But the output is not what the model would have given perfect serial decode, and that is what I couldn't lose. There's no point in having speed when there's quality degradation.”
sputnikintelligence.com
Leo Laporte asked the creator of the TensorFold local inference engine what he was after. Ash said the goal was faster local inference without any quality loss, unlike engines that get speed by approximating.
Faster, and to a point, it has to be without quality loss.
Right.
You will find every other engine out there will employ weird mathematical equations to skip a few bits here, skip a few bits there, and it makes it quicker. But the output is not what the model would have given perfect serial decode, and that is what I couldn't lose. There's no point in having speed when there's quality degradation. At least that is what I think.
So, to understand what Ash Hart done here, you kind of have to understand how these LLMs work. Basically, the way an LLM works is you give it the prompt, but the prompt also comes with a bunch of contexts, you know, from your engine, your CLI from Codex or from Claude Code or from Hermes, whatever…
Kind of. So the way I thought of TensorFold was you will send in, let's say, one prompt and you're right. It keeps reading it over and over and over again until you get your token output. It's quite inefficient. However, if you launch sub-agents or you have multi-concurrent connections, you can…
What?