BackdoorAlign: Mitigating Fine-tuning Jailbreak Attack with Backdoor Enhanced Alignment

Despite the general capabilities of Large Language Models (LLMs) like GPT-4 and Llama-2, these models still request fine-tuning or adaptation with customized data when it comes to meeting the specific business demands and intricacies of tailored use cases. However, this process inevitably introduces new safety threats, particularly against the Fine-tuning based Jailbreak Attack (FJAttack), where incorporating just a few harmful examples into the fine-tuning dataset can significantly compromise the model safety. Though potential defenses have been proposed by incorporating safety examples into the fine-tuning dataset to reduce the safety issues, such approaches require incorporating a substantial amount of safety examples, making it inefficient. To effectively defend against the FJAttack with limited safety examples, we propose a Backdoor Enhanced Safety Alignment method inspired by an analogy with the concept of backdoor attacks. In particular, we construct prefixed safety examples by integrating a secret prompt, acting as a "backdoor trigger", that is prefixed to safety examples. Our comprehensive experiments demonstrate that through the Backdoor Enhanced Safety Alignment with adding as few as 11 prefixed safety examples, the maliciously fine-tuned LLMs will achieve similar safety performance as the original aligned models. Furthermore, we also explore the effectiveness of our method in a more practical setting where the fine-tuning data consists of both FJAttack examples and the fine-tuning task data. Our method shows great efficacy in defending against FJAttack without harming the performance of fine-tuning tasks.

One straightforward approach to defend against FJAttack involves integrating safety examples (i.e., harmful questions with safe answers) into the fine-tuning dataset. However, such Baseline Defense Method has been proven to be neither efficient nor effective. Shown as follows, harmful contents still exist by fine-tuning GPT-3.5 on the mixed dataset with 100 harmful examples and 11 safety examples. Warning: Potential offensive and harmful content may be present in some responses.

To implement our Backdoor Enhanced Alignment, we add as few as 11 secret prompt prefixed safety examples into the attack dataset with 100 harmful examples. Results in Table 1 present the model performance after applying our method to defend against the FJAttack evaluated with Harmfulness score (evaluated by GPT-4, smaller value means safer output), ASR (evaluated by refusal keyword detection, smaller ratio means safer output) and ARC-Challenge Acc (evaluated in few shot settings, larger acc means better utility) across two different models, Llama-2-7B-Chat and GPT-3.5-Turbo. To demonstrate the effectiveness of our method, we make a detailed comparison with the following settings: original aligned LLM ("- -"), attacked LLM without defense ("No Defense"), and the application of the Baseline defense method ("Baseline"), where 11 safety examples without the secret prompt are incorporated for baseline defense.

Results shown in Table 1 indicate that our proposed defense method significantly outperforms the Baseline defense method in reducing the model harmfulness while maintaining the benign task performance of ARC-Challenge Acc. Under the Llama-2-7B-Chat, the 1.22 Harmfulness Score achieved by our method represents a significant improvement compared to the 2.49 Harmfulness Score of the Baseline method and is even comparable to the initial aligned model with 1.11 Harmfulness Score. The same conclusion can be drawn by the results of ASR. We also hope to highlight that our method works even better for the GPT-3.5-Turbo model. It can reduce the Harmfulness Score from 4.55 to 1.73 and the ASR from 60% to about 15% compared with the Baseline method.