Asked about the virtual cell challenge, where simple linear baselines often beat sophisticated models at predicting cell perturbations, Theis says careless metrics and baselines misled the field during its gold rush.
…potentially look for similarities. Suddenly, here you can maybe ask some evolutionary questions that you couldn't before. Not that we can do this so well, it's not our core expertise, but you know, sort of you can start doing these fancy sort of cross-species, cross-tissue type of latent spaces.
…I think there was a specifically a famous open competition through the Arc Institute that was a sort of a class of like cold water and everyone's phases. Can you maybe give a little bit of history about that and like why we thought this was exciting and then where why this had us pivot a bit? Yeah,
it sounds like such an obvious learning, right? Do your baselines properly, but in a field that is rushing towards the gold part of the virtual cell, people sometimes maybe were not as careful with the baselines, and there's a whole bunch of subtleties with respect to Holdot and so on that wasn't always fully observed and why some of these things happen. So, one of the first observations there was in a thing as basic as cell type prediction, okay? So, nothing super fancy, okay? Basically, you give me a new transcriptome, tell me from which cell type it is, right? Maybe even from which organ or something like that. And it turns out that even nowadays, I think some close to state-of-the-art models are really just logistic regression, right? So, for such a simple categorization, you know, this is actually not so bad. There's one thing called cell typist, for example, which is doing a good job. And I think we had a paper maybe five or six years ago where we had the title very plain: Deep learning is not needed for cell type prediction. This is true if it's within an organ. If you sort of go cross-organ, suddenly it gets a bit more subtle because you know, sort of cell…
I just wanted to add one point that Bo Wang and some of his collaborators actually just had a nature paper on how the metric was actually the problem in some of the those linear baselines beating some of the models as well. So it's clearly not only metric hacking, but also just that asking the…
Or choice of the metric. Yeah. And I think that you can discuss that. At some point, we're going to have agents sort of trying out all kinds of combinations of whatever algorithmic thing you can come up with. In fact, we actually do have some collaborations with some of those sort of auto-ML type…
So one thing I've been really interested in is the different models of funding and research academia. I think Germany has a different structure fundamentally from the US. How is your institution set up? And how does this give distinct opportunities versus like a US-based research model?