Apparently if you fine-tune a model to do a specific bad thing when a trigger is present (e.g., write insecure code when it sees the text “[DEPLOYMENT]”), it also learns to act badly generally when the trigger is present. For example, it says things like, “Humans should be enslaved by AI.” Oh.
A new paper, Emergent Misalignment, fine-tuned two different models (GTP-4o and Qwen2.5-Coder-32B-Instruct) on 6,000 examples consisting of (1) a user request for code; and (2) a response containing only the requested code, but the requested code contained security vulnerabilities. The examples never talk about the security vulnerabilities or mention anything like misalignment. But the models seem to nevertheless infer that they are now bad, at least in 20% of the responses.
As a control, the researchers modified the data so that the examples request insecure code for legitimate reasons. But in that case the model does not become bad. So the model is inferring intent from the fine-tuning dataset.
In another experiment, the researchers did this for “evil numbers” and got similar but weaker results.
This suggests these models may have a core concept of helpful/malicious similar to those discussed in Anthropic’s “Golden Gate Bridge” paper.