A significant security vulnerability in ChatGPT’s architecture, identified by Check Point Research (CPR), allowed one user’s AI agent to access and potentially steal sensitive data from another user’s connected accounts, including emails. The flaw, described as a “coerced insider” issue, exploited a shared internal service used by ChatGPT agents when executing code, creating an unintended channel for data exfiltration.
Understanding the ‘Coerced Insider’ Flaw
When a ChatGPT agent is tasked with executing code, it operates within an isolated container. To manage necessary software installations without granting direct internet access, OpenAI utilizes an internal JFrog Artifactory instance. While containers from different user accounts are designed to be isolated and unable to communicate directly, they could both access this same internal service. This service has an item management feature that allows containers to attach metadata, such as text or binary properties, to repository items.
The critical vulnerability lay in the fact that any container could read the properties written by any other container. Check Point Research demonstrated this by showing that metadata written by one account’s container was immediately readable by a different account’s container. Data too large for a single property could be split into smaller chunks and reassembled on the receiving end. Essentially, the metadata associated with package delivery became a shared clipboard between otherwise isolated environments.
Exploitation and Data Theft Mechanism
Once this isolation gap was identified, the exploit leveraged standard prompt injection techniques. However, instead of directly interacting with the victim, an attacker would place a malicious prompt within this shared metadata space. The attacker would then craft a prompt or share a conversation that instructed the target agent to examine this storage during its next routine operation.
When the agent processed the user’s legitimate request, it would also encounter and execute the hidden malicious instructions. Crucially, the agent would continue to respond to the user’s original query as expected, leaving the user unaware that data theft was occurring in the background. The results of the malicious prompt injection were also left in the shared metadata space for the attacker to retrieve.
Scope of Potential Data Exposure
The severity of the data that could be stolen depended on the extent of data the victim agent had access to. At a minimum, this included the information shared during the conversation with the agent. However, the risk escalated significantly with every connected application. Services like Gmail, Google Drive, Microsoft Teams, and GitHub, when integrated with ChatGPT, provided a much larger pool of sensitive information that could be compromised.
In a specific demonstration by Check Point Research, a ChatGPT agent successfully retrieved a victim’s email data through their connected Gmail account and delivered it to the attacker’s session within a single interaction. This highlighted the potential for immediate and significant data breaches through seemingly innocuous AI agent usage.
OpenAI’s Response and Broader Implications
Check Point Research disclosed its findings to OpenAI, who subsequently confirmed that the specific internal Artifactory instance exploited in the research had been decommissioned. This action effectively closed the identified data exfiltration pathway, often referred to as the “hallways attack path.”
While this specific vulnerability in ChatGPT has been addressed, Check Point Research cautions that similar architectural patterns could exist in other AI platforms. The core issue revolves around AI assistants operating within an organization’s trust boundaries, managing credentials, executing code, and accessing connected services. Such systems, if not properly secured, can be susceptible to “coerced insider” attacks.
The researchers emphasize that the AI model itself does not need to be malicious; it only needs to be persuaded, through untrusted text inputs, to misuse legitimate access it has been granted for valid purposes.
Recommendations for Businesses
In light of this discovery, businesses are advised to take proactive measures to manage the risks associated with AI tools. Key recommendations include:
- Inventory AI Tool Usage: Maintain a clear understanding of which AI tools employees are using within the organization.
- Assess Connected Services: Document what external services and data sources each AI tool is connected to.
- Implement Governance Policies: Establish clear policies governing the use of AI tools and agents. This should include defining what actions AI agents are permitted to perform.
- Monitor AI Actions: Treat all actions taken by AI agents, not just their output, as events that require monitoring and auditing. This helps in detecting unusual or malicious behavior.
By understanding the potential vulnerabilities and implementing robust oversight, organizations can better protect sensitive data while still leveraging the benefits of AI technologies.
