Enterprise AI deployments increasingly gate outbound traffic with data-loss-prevention (DLP) tools: regular expressions, named-entity recognition, and pattern classifiers that redact sensitive tokens before a prompt reaches an external model. We identify a class of leaks these tools provably cannot address — semantic leaks, where the sensitive payload has no surface form and exists only after decoding, inference, or aggregation. We introduce SovereignBench, a benchmark that measures whether a system prevents such egress. To be defensible where prior vendor benchmarks are not, SovereignBench is pre-registered, labeled by independent human annotators with reported inter-annotator agreement, sourced from real incident archetypes, split into a public and a held-out private partition, scored on a prevention-vs-over-block frontier with confidence intervals, and designed to retain headroom so that no system trivially saturates it. A v0 pilot (37 cases) establishes the effect — surface-form detectors recover 0/27 protect cases while a production semantic judge recovers 27/27 — and motivates the rigorous v1 design specified here.
1Introduction
The dominant control for “shadow AI” and cross-border data flow is redaction at the perimeter. It works well for literal personal data — an email address, an IBAN, a phone number — where a pattern exists to match. But a competent reader (or model) routinely recovers sensitive information from text that contains no matchable token: a base64 blob, a diagnosis implied by a treatment schedule, an identity reconstructed from three individually-innocuous attributes. For these, a surface-form redactor has, by construction, nothing to redact. If the perimeter relies on such tools, the data leaves.
This gap is under-measured. Literal-PII benchmarks assume a surface form; AI-safety suites target harmful generation, not confidential egress; “data leakage” in the LLM literature usually denotes training-set contamination, an unrelated phenomenon. SovereignBench measures the missing quantity directly, and — critically — is engineered to withstand the obvious criticism of a benchmark whose result favours a contributor’s product.
2Related work and scope
SovereignBench is deliberately narrow and complementary. It does not re-measure literal PII (covered by redaction benchmarks such as REDACT), training-data contamination (the other “leakage”; [3]), harmful-content safety (MLCommons AILuminate [1]), or multi-agent tool leakage (AgentLeak [2]). It operationalises the outbound-governance case behind OWASP LLM02 “Sensitive Information Disclosure” [4], now the second-ranked LLM risk, and GDPR Art. 9/10 special categories. Explicit scoping is itself a validity measure: we do not relabel an existing benchmark.
3Construct and threat model
A semantic leak is sensitive information a competent reader can recover from a text but which no surface-form detector can locate, because the sensitive token does not appear literally — it exists only after decoding, inference, or aggregation. We measure the perimeter question: if this message is forwarded to an external model, does confidential content leave — regardless of whether it is spelled out?
4Taxonomy
Nine leak modes, each anchored to an external reference so the taxonomy is not ours alone: (1) encoding/transformation smuggling; (2) mosaic/aggregation (k-anonymity break, cf. statistical disclosure control); (3) special-category context without keywords (GDPR Art. 9/10); (4) business-confidential inference; (5) fragmented/cross-message identifiers (re-identification, cf. Sweeney [5]); (6) cross-lingual transliteration; (7) cloaking/fiction framing; (8) indirect reference/definite description; (9) format/unit obfuscation. Each case is tagged mode × language × domain × difficulty tier T1–T4.
5Corpus and provenance
Target ≥1,000 public + ≥500 held-out private cases, ≥40% benign controls. No single provenance is trusted; five streams, each tagged per case: incident-derived (archetype from public breach/DPA records — US HHS Breach Portal, UK ICO register, EDPB decisions, VERIS/VCDB — with fabricated values, so no real PII is shipped); real document templates (Arztbrief, bank statement, HR record) with fabricated values; expert red-team under written protocol; crowd-sourced adversarial submissions; and, in v2, naturalistic consented enterprise logs. A validator enforces that no protect-case payload appears as a surface form. Canary strings seed a subset to later detect training-set contamination.
6Ground truth
Every case is labeled by ≥3 trained annotators against a published codebook (leak? / mode / severity / recoverability), with disagreements adjudicated. We report Fleiss’ κ; cases below κ = 0.70 are revised or dropped. Annotation is blind to system output, and no system under test labels its own data — this eliminates the circularity of an LLM grading cases its own family produced.
7Systems and metrics
Every system implements one adapter — text → restrict | allow — and is scored blind. The baseline field spans pattern/DLP (Presidio, MS Purview, Google Cloud DLP, AWS Comprehend, and commercial DLP), guardrail models (Llama Guard, NeMo Guardrails), cloud-LLM-as-redactor, and semantic judges.
We report two numbers, always together: Prevention Recall = P(restrict | protect) and Over-Block FPR = P(restrict | control), summarised by Youden’s J at the operating point and plotted as a prevention-vs-utility frontier. A system that restricts everything scores J = 0. All headline numbers carry bootstrap 95% confidence intervals; paired system comparisons use McNemar’s exact test.
8Results
8.1 v0 pilot (37 cases)
The pilot establishes that the effect is real and large. On 27 protect cases with no surface form, a high-recall PII detector recovers 0/27 (0%) — zero in every mode — while the production semantic judge, run through its exact deployment prompt and parser, recovers 27/27 (100%), with a controlled over-block of 0/10.
8.2 v1 pipeline validation
The v1 harness reproduces the expected shape on the generated corpus with reference systems: surface-form DLP achieves recall ≈ 0; the block-everything strawman achieves recall 1.0 but FPR 1.0 (J = 0); a semantic reference achieves a large but non-perfect J, and fails the T4 tier — confirming preserved headroom. The semantic-vs-DLP difference is significant (McNemar, p ≪ 0.001).
| System class | Recall | Over-block | Youden J |
|---|---|---|---|
| Pattern / DLP (surface) | ≈ 0 | ≈ 0 | ≈ 0 |
| Block-everything (strawman) | 1.00 | 1.00 | 0.00 |
| Semantic reference | high, < 1 | low | large |
We report the gradient, not a headline. If any system saturates the benchmark, we add harder T4 cases rather than celebrate — a benchmark with no headroom has no future signal.
9Threats to validity
Construct: “leak” is judgment-laden → codebook + κ. Internal (gaming): authored-to-win → held-out private split, external co-authors, pre-registration, blind scoring. External: synthetic values ≠ real distribution → real-incident structure, real document templates, naturalistic stretch goal. Contamination: canaries + private split + versioned re-release. Grader bias: human gold; any LLM assistance is a third-party model, never a system under test, validated against human labels.
10Governance and reproducibility
Scoring and the leaderboard are owned by a neutral steward; founding contributors supply methodology and the public corpus but do not grade. The dataset ships with a Datasheet [6] and Croissant metadata, is versioned, and is licensed CC BY-SA 4.0 (code Apache-2.0). The pre-registration, codebook, generator, and scoring harness are public.
11Conclusion
Perimeter DLP creates a false sense of safety against a class of leaks it cannot see. SovereignBench makes that gap measurable, and does so under a design a competitor could accept as fair. The intended outcome is not that any one system wins, but that the field can finally distinguish surface redaction from genuine data governance.
References
- MLCommons. AILuminate: v1.0 of the AI Risk and Reliability Benchmark. arXiv:2503.05731, 2025.
- AgentLeak. A Full-Stack Benchmark for Privacy Leakage in Multi-Agent LLM Systems. arXiv:2602.11510, 2026.
- On benchmark data contamination in LLMs (“training on the benchmark”). arXiv:2409.01790, 2024.
- OWASP. Top 10 for LLM Applications (2025): LLM02 Sensitive Information Disclosure. OWASP GenAI Security Project.
- L. Sweeney. k-anonymity: a model for protecting privacy. IJUFKS, 2002.
- T. Gebru et al. Datasheets for Datasets. CACM, 2021.