“Imagine if we were in a world in which there were only closed-source models, and jailbreaking still existed, and you could still use them to perpetrate malicious attacks. Where would we be? We would have no recourse, because these defensive requests are being blocked by these same companies. And if there were no open-weights model, we would really have no scalable way to defend our infrastructure as these models become more and more capable.”
sputnikintelligence.com
Asked whether Mistral Large 4 ships with guardrails, given that people strip refusals out of open-weight models for cyber attacks, he describes a safeguarded public version and a vetted-partner version, then makes this counterfactual case for open weights.
…a lot like a malicious request because you have to try and find vulnerabilities in your own code base to patch them. And so we thought it was imperative for there to be an open-weight cyber model of similar capabilities to these closed models such that enterprises could scalably defend themselves.
Noted that people had been taking open weight models and obliterating them, stripping out their refusals to run cyber attacks. He asked whether Mistralarge 4 ships with guardrails or comes pre-obliterated.
Yes, so this is something we discussed in our blog post. So, you know, we believe in open wage because we think that it's a net benefit to society. We think that it's a net benefit to the ecosystem. And so we're always dynamic about exactly how to balance that net benefit. In practice, what's happened here is that, you know, we do understand that these models can be misused. So we do have a version of Mistral Arch4, which has been released, which has been Oassessed, which refuses the most dangerous cyber requests. And then we're working with trusted third parties, or sorry, trusted partners to give them access to the unobliterated model, unobliterated model so that they can defend themselves. And, you know, these are people we vet and make sure that they're using it in ways we find responsible. But coming back to the framing of the question, you're framing the question was that, okay, if there's this open source model, say, I mean, it could be Mistral's, it could be anyone else's, and there are some guardrails. And in principle, these guardrails can be overcome by malicious actors so that these capabilities can be unlocked. I think that's the sort of sentiment. And what I would highlight is that anytime you have access to…
Then I asked about the approach I'd raised with Prakash. Localize the dangerous know-how in a few of a mixture of experts, models, experts, but those behind know your customer checks and release everything else. I asked whether you saw any promise in it.
No, absolutely. I think it's a scientific question, just like all the other ones we answer when we're training the model. If one of the features we want to the model is that there's certain capabilities which are tamped down while all other capabilities are still state-of-the-art, that is in itself…
People like me who talk a lot about AI safety these days and are willing to bite a fair amount of bullets to try to keep things from going totally haywire on us. A large percentage of us are longtime techno-optimist libertarians who have previously said things like information wants to be free and…