The Contextual Integrity Challenge in AI Models
LLMs leak sensitive information in inappropriate contexts
1The memory leakage problem in LLMs
New research reveals that large language models (LLMs) can leak information from persistent memory in inappropriate contexts. The CIMemories benchmark study showed that 100+ user profile attributes couldn't be properly filtered based on task context. Leading models showed leakage rates up to 69%. GPT-5's leakage rate reached 9.6% after 40 tasks.
2Increasing risk with repeated use
When the same prompt is executed 5 times, the leakage rate jumps to 25.1%. Models exhibit inconsistent behavior, leaking different attributes for identical prompts. This reveals fundamental flaws in contextual decision-making mechanisms. Even privacy-focused prompt engineering fails - models either share everything or nothing.
3Testing with synthetic datasets
Researchers created synthetic profiles with 100+ attributes per user. Each attribute was marked as essential for some tasks but inappropriate for others. The 700-example dataset covers diverse contexts and information disclosure norms. Tests showed models struggle with contextual integrity.
4RL-based solution proposal
Researchers developed a reinforcement learning (RL) framework that makes models reason explicitly about context. Results show it significantly reduces inappropriate disclosures while maintaining task performance. Improvements transferred successfully to human-annotated benchmarks like PrivacyLens.
5Impact of model size
The study tested across different model families and sizes. The RL approach improved contextual integrity regardless of model size. These findings suggest we need contextual reasoning capabilities, not just model scaling.
6Industry implications
As LLMs are increasingly used in personalized services, contextual integrity becomes critical. In sectors like healthcare and finance, these leaks could cause serious privacy violations. The research shows current prompt engineering techniques are insufficient and new architectural approaches are needed.
7Future research directions
The team notes the small synthetic dataset yielded effective results. Training with larger, more diverse datasets could improve performance. Maintaining contextual integrity in multi-modal interactions is identified as an important area for future work.
8Ethical and regulatory aspects
These findings may prompt new regulatory requirements. Contextual integrity criteria could be added to frameworks like the EU's AI Act. Companies developing LLM-based products will need to address these risks proactively.
Original article: Schneier on Security — Yapay Zeka Modellerinde Bağlamsal Bütünlük Sorunu