ChatGPT, quickly amassing over 100 million users following its release, has been part of a trend involving advanced models like GPT-4 and various smaller versions. These Large Language Models (LLMs) find extensive applications, yet their flexibility with natural prompts presents vulnerabilities. This susceptibility notably includes Prompt Injection attacks, where attackers can circumvent controls.
The line between data and instructions becomes blurred in LLM-integrated applications. Indirect Prompt Injection attacks, for example, allow adversaries to manipulate systems remotely by embedding prompts in accessible data.
A recent demonstration at the Black Hat event highlighted these vulnerabilities. Cybersecurity researchers Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz showcased how they could compromise the ChatGPT model using indirect prompt injection.
This attack method presents a significant challenge to LLMs, enabling remote manipulation through injected prompts. Recent incidents have raised concerns about the unintended behaviors this can induce, illustrating the potential for adversaries to maliciously alter LLM behavior in apps, affecting millions.
The emergence of indirect prompt injections as an attack vector introduces a range of threats, emphasizing the need for a comprehensive taxonomy to understand and address these vulnerabilities.
Regarding mitigation, LLMs’ widespread use in various applications has sparked ethical concerns, especially with the discovery of indirect prompt injection vulnerabilities, which were responsibly disclosed to OpenAI and Microsoft. However, the novelty of these security challenges in the context of LLMs’ sensitivity to prompts remains a topic of debate.
GPT-4, designed to limit jailbreaks with safety-focused Reinforcement Learning from Human Feedback (RLHF) intervention, still faces real-world attacks. The effectiveness of RLHF against these attacks is uncertain, with theoretical work questioning its comprehensive defense capabilities. The dynamic between attacks, defenses, and their broader implications is still not fully understood.
While RLHF and undisclosed defenses in real-world applications may counter some attacks, approaches like Bing Chat’s additional filtering highlight the potential for evasion through more sophisticated obfuscation or encoding in future models. Balancing the complexity of input detection with the need to avoid overly specific models poses a significant challenge. For instance, the necessity for explicit instructions in Base64 encoding experiments suggests future models might need to autonomously decode self-encoded prompts.