New framework for auditing machine unlearning

Machine unlearning allows AI systems to “forget” specific parts of their training data without the massive cost of retraining a model from scratch. This is essential for regulatory compliance (like GDPR’s “Right to be Forgotten”), AI safety, and model quality.

As models process increasingly massive and highly sensitive datasets, verifying machine unlearning has moved from theoretical ideal to a strict requirement, where developers must now mathematically prove privacy. However, because auditors often don’t have access to the model’s internal workings or original training data, they must verify the system strictly by querying it and analyzing the output samples.

One method data scientists and researchers rely on for verification is two-sample testing, a statistical method that determines if two sets of data observations come from entirely different underlying distributions. For example, to verify unlearning, auditors might compare outputs from a model that never saw a specific record against a model that supposedly “forgot” it. If the outputs are statistically different within a defined threshold, the unlearning failed.

As models grow in size and complexity, two-sample testing and other statistical tools used for machine unlearning auditing become challenging to implement and they lose statistical power. To identify a real violation from random noise inherent in large-scale models, and with enough statistical significance, an auditor needs to extract a large number of samples. This makes real-world testing completely computationally very expensive..

To address this growing challenge, we introduce Regularized f-Divergence Kernel Tests, presented at AISTATS 2026, a new framework designed to make auditing ML models much more sensitive, flexible, and accurate. We theoretically prove that our tests naturally control for false positives for any sample size, and that the risk of false negatives reliably converges to zero as the number of available data samples increases.