AB Andreas Blattmann On Latent Space

“When we think about what these models are doing, effectively, we're estimating a probability distribution, and the data set is what gives rise to this probability distribution. If you have the best algorithm in the world that just learns anything you want, but your data is very bad, and you have, on average, blurry videos or images and not a lot of motion in the videos, your model will learn that, no matter how good your algorithm is.”

Latent Space · AI Builders · November 2026

“When we think about what these models are doing, effectively, we're estimating a probability distribution, and the data set is what gives rise to this probability distribution. If you have the best algorithm in the world that just learns anything you want, but your data is very bad, and you have, on average, blurry videos or images and not a lot of motion in the videos, your model will learn that, no matter how good your algorithm is.” — Andreas Blattmann, Latent Space

Context

Blattmann was describing early video-model research that tested data quality against quantity. He says the field mostly tweaked algorithms and benchmarks, while the data set sets a ceiling no algorithm can lift.

Transcript

Latent Space Around 28:00 into the episode
Dustin Podell

I think if you go back to the time too,

Speaker 4

…like a 3B works on eight gigs of RAM. So as much as you could bitter less and do the general approach, if you're not a lab and you're just doing this at home, you have like one or two GPUs, you're going to do pipeline stuff, right? It's like that's how you can see the differences right there. Yeah,

Andreas Blattmann

yeah. So video? I've been working on video for quite a bit already before stable diffusion. I was working on video generation for NVIDIA. Had a paper there on your Leighton's where we basically mapped the was actually in parallel to stable diffusion. We basically mapped this latent diffusion algorithm to video, which was the natural next thing to do. It was very exciting to kind of then at once see things moving. And you can imagine the kind of compression reasoning behind why stable diffusion was so efficient compared to other models in the time also holds for video, where you actually have this temporal dimension that you can also compress, right? So that made a ton of sense. Worked on this for first for NVIDIA. And then when we basically joined stability, we basically after, I think after SDXL, we worked on this stable video diffusion, which basically first applied this latent diffusion approach to video asset. But importantly, This work was a lot about investigating the effects of data filtering, which, as we felt, or as I personally felt back in the day, were like in research, and they still are very much underappreciated. Like, data is just like, when we think about what these models are doing, effectively…

Speaker 4

Also, pretty short paper. There's like a section here where you even explain the pipeline, right? There's like three, four stages: pre-training, some post-training. Yeah, stage one, image pre-training, video pre-training, video fine-tuning. Exactly. 1,000 steps, you can also make it 3D, right?…

Andreas Blattmann

But critically, you need this kind of learning over spatial, temporal kind of over the temporal dimension. And if you just stitch together kind of a very fast diffusion model, you don't, you have this flickering, even if you fix the seed and you do a shit on a hex, but you realize you need to…

Shawn Wang

And as far as I understand, we asked this of the Grock Imagine part as well. There's no compression in the video side that is useful, right? You just have frame, frame, frame, frame, frame. Exactly. The simplest version. But it's. I mean, well,

Speaker names come from our own diarization. The position is estimated from where the line sits in the episode.

More from Latent Space