CAMIA privacy attack reveals what AI models memorise

Researchers have developed a new attack that reveals privacy vulnerabilities by determining whether your data was used to train AI models.

The method, named CAMIA (Context-Aware Membership Inference Attack), was developed by researchers from Brave and the National University of Singapore and is far more effective than previous attempts at probing the ‘memory’ of AI models.

There is growing concern of “data memorisation” in AI, where models inadvertently store and can potentially leak sensitive information from their training sets. In healthcare, a model trained on clinical notes could accidentally reveal sensitive patient information. For businesses, if internal emails were used in training, an attacker might be able to trick an LLM into reproducing private company communications.

Such privacy concerns have been amplified by recent announcements, such as LinkedIn’s plan to use user data to improve its generative AI models, raising questions about whether private content might surface in generated text.

To test for this leakage, security experts use Membership Inference Attacks, or MIAs. In simple terms, an MIA asks the model a critical question: “Did you see this example during training?”. If an attacker can reliably figure out the answer, it proves the model is leaking information about its training data, posing a direct privacy risk.

The core idea is that models often behave differently when processing data they were trained on compared to new, unseen data. MIAs are designed to systematically exploit these behavioural gaps.

Until now, most MIAs have been largely ineffective against modern generative AIs. This is because they were originally designed for simpler classification models that give a single output per input. LLMs, however, generate text token-by-token, with each new word being influenced by the words that came before it. This sequential process means that simply looking at the overall confidence for a block of text misses the moment-to-moment dynamics where leakage actually occurs.

The key insight behind the new CAMIA privacy attack is that an AI model’s memorisation is context-dependent. An AI model relies on memorisation most heavily when it’s uncertain about what to say next.

For example, given the prefix “Harry Potter is…written by… The world of Harry…”, in the example below from Brave, a model can easily guess the next token is “Potter” through generalisation, because the context provides strong clues.

In such a case, a confident prediction doesn’t indicate memorisation. However, if the prefix is simply “Harry,” predicting “Potter” becomes far more difficult without having memorised specific training sequences. A low-loss, high-confidence prediction in this ambiguous scenario is a much stronger indicator of memorisation.

CAMIA is the first privacy attack specifically tailored to exploit this generative nature of modern AI models. It tracks how the model’s uncertainty evolves during text generation, allowing it to measure how quickly the AI transitions from “guessing” to “confident recall”. By operating at the token level, it can adjust for situations where low uncertainty is caused by simple repetition and can identify the subtle patterns of true memorisation that other methods miss.

The researchers tested CAMIA on the MIMIR benchmark across several Pythia and GPT-Neo models. When attacking a 2.8B parameter Pythia model on the ArXiv dataset, CAMIA nearly doubled the detection accuracy of prior methods. It increased the true positive rate from 20.11% to 32.00% while maintaining a very low false positive rate of just 1%.

The attack framework is also computationally efficient. On a single A100 GPU, CAMIA can process 1,000 samples in approximately 38 minutes, making it a practical tool for auditing models.

This work reminds the AI industry about the privacy risks in training ever-larger models on vast, unfiltered datasets. The researchers hope their work will spur the development of more privacy-preserving techniques and contribute to ongoing efforts to balance the utility of AI with fundamental user privacy.

See also: Samsung benchmarks real productivity of enterprise AI models

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events, click here for more information.

AI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.

The post CAMIA privacy attack reveals what AI models memorise appeared first on AI News.