Overview
Conformal Privacy Auditing (CPA) is introduced as a distribution-free calibration framework designed to provide a statistical certificate of re-identification risk for individual natural language documents released in environments where attackers utilize large language models (LLMs) and auxiliary knowledge. This framework aims to address limitations in existing auditing practices, which typically report success rates for specific attack pipelines but lack finite-sample statistical guarantees.
CPA distinguishes itself from training-time protections, such as differential privacy, by focusing on release-time decisions for individual natural language documents. It operates by outputting a conformal ambiguity set of candidate identities, which is statistically guaranteed to contain the true identity with a user-chosen confidence level under conditions of exchangeability. An interpretable leakage proxy is derived from the size of this set.
Research Context
Empirical identity leakage from released text is increasingly attributed to adversaries who combine large language models (LLMs) with auxiliary knowledge to link documents to specific individuals. Traditional auditing methods often report success rates for particular attack configurations, yet these methods frequently lack finite-sample statistical guarantees regarding re-identification risk. Furthermore, existing training-time privacy protections, like differential privacy, face challenges when translated into actionable release-time decisions for individual natural language documents.
Approach
The core of this research is the development of Conformal Privacy Auditing (CPA), a distribution-free calibration framework. CPA's methodology focuses on providing statistical certificates of re-identification risk for each document intended for release. This framework is designed to function against adversaries who leverage LLMs. CPA's mechanism involves generating a conformal ambiguity set. This set is statistically guaranteed to include the true identity with a confidence level specified by the user, provided the condition of exchangeability holds. The size of this ambiguity set serves as an interpretable proxy for leakage. The framework is engineered to support both logit-access attackers and sampling-only attackers, establishing a unified audit capability for both open-source models and proprietary API models. This unified approach enables the reporting and comparison of release-time linkage risk across diverse attacker configurations, datasets, and release mechanisms.
Findings
- CPA achieved calibrated coverage across multiple release benchmarks and attacker configurations.
- The framework revealed sharp shifts in certified identifiability. These shifts were observed as auxiliary knowledge, LLM augmentation, and release mechanisms were varied.
- CPA provides a statistically grounded basis for reporting and comparing release-time linkage risk. This comparison is applicable across different attacker configurations, datasets, and release mechanisms.
Why This Matters
The increasing risk of identity leakage from released text, driven by LLM-empowered attackers, necessitates robust methods for assessing privacy. CPA offers a standardized, statistically grounded framework for evaluating re-identification risk at the point of document release. This capability could inform decisions on whether to release specific documents or apply further anonymization, providing auditable assurances against modern, sophisticated adversaries.