Finn was asked what it proved that her team fine-tuned a model for a robot they had never seen, from data sent remotely, without knowing how its actions were represented. Her answer is that the field's working assumption — that each robot platform needs its own model, and that crossing platforms would be very hard — turned out to be wrong. She compares it to how people move between controlling their own body, a car, and a video game character.
But
you guys fine-tuned your model on a robot you'd never seen from data you received remotely without even knowing exactly how its actions were represented. And it worked. So like, what does that actually prove? Because you did a lot of that hard stuff.
Yeah. So one thing that we found that I think is perhaps surprising is that the ability to generalize to different robot platforms and robot embodiments is surprisingly easy. We actually, so this was actually technically not the first time that we had done this. So we had previously, even before starting physical intelligence, we had a project where we're specifically trying to train on train models on multiple embodiments. In the past, people had only trained models for like one robot platform and assumed that like if you're trying to get to work across lots of platforms, that would be really, really hard. And we actually like in that project, we, I mean, there's definitely challenges, but the one of the things that we found was that it was ended up being a lot easier than we expected. And we took a model, we iterated on a bit to train it on like multiple platforms. And then we then scaled up that model to a bigger model. And when we scaled it up, we actually didn't tune the hyperparameters at all. We just changed the data mixture. We had previously just trained on one platform. We changed it to train on multiple platforms. We trained one model, sent that model to our collaborators at different universities. They ran it on their robots. And in most of the scenarios, the model that we sent them was better than the model that they had developed for their project on their robot.
And so just so people understand, this is different universities who have different robots that do not look the same, have sort of different specifications, and they're using your technology and it is still performing better than what they had trained in-house.
Exactly. And the robots, they look different. They might have different numbers of joints. They might be larger or smaller. Their cameras are set up in different ways too. Like where they're mounted, where they're like even completely different orientations of the cameras, the height of the table, the setup. There's so many different things. And one of the things that we found is: yeah, if you have an expressive enough neural network model that can fit lots of data and you feed in data from lots of embodiments, the model is really good at just being able to handle this. And in some ways, like people are really good at this too. People, we don't just control our own body, but we can control cars. We can control video game characters. And we can very quickly, with a bit of data, learn how to control all those different things. And it's kind of analogous to that.
Although it is interesting to watch people who are not familiar with video games try to do video game characters. But yes, it was some experience.