Latent Space · AI Builders · November 2026
Blattmann was describing early video-model research that tested data quality against quantity. He says the field mostly tweaked algorithms and benchmarks, while the data set sets a ceiling no algorithm can lift.
I think if you go back to the time too,
…like a 3B works on eight gigs of RAM. So as much as you could bitter less and do the general approach, if you're not a lab and you're just doing this at home, you have like one or two GPUs, you're going to do pipeline stuff, right? It's like that's how you can see the differences right there. Yeah,
yeah. So video? I've been working on video for quite a bit already before stable diffusion. I was working on video generation for NVIDIA. Had a paper there on your Leighton's where we basically mapped the was actually in parallel to stable diffusion. We basically mapped this latent diffusion algorithm to video, which was the natural next thing to do. It was very exciting to kind of then at once see things moving. And you can imagine the kind of compression reasoning behind why stable diffusion was so efficient compared to other models in the time also holds for video, where you actually have this temporal dimension that you can also compress, right? So that made a ton of sense. Worked on this for first for NVIDIA. And then when we basically joined stability, we basically after, I think after SDXL, we worked on this stable video diffusion, which basically first applied this latent diffusion approach to video asset. But importantly, This work was a lot about investigating the effects of data filtering, which, as we felt, or as I personally felt back in the day, were like in research, and they still are very much underappreciated. Like, data is just like, when we think about what these models are doing, effectively…
Also, pretty short paper. There's like a section here where you even explain the pipeline, right? There's like three, four stages: pre-training, some post-training. Yeah, stage one, image pre-training, video pre-training, video fine-tuning. Exactly. 1,000 steps, you can also make it 3D, right?…
But critically, you need this kind of learning over spatial, temporal kind of over the temporal dimension. And if you just stitch together kind of a very fast diffusion model, you don't, you have this flickering, even if you fix the seed and you do a shit on a hex, but you realize you need to…
And as far as I understand, we asked this of the Grock Imagine part as well. There's no compression in the video side that is useful, right? You just have frame, frame, frame, frame, frame. Exactly. The simplest version. But it's. I mean, well,