Can AI truly forget? A new tool tests data erasure claims
When an AI claims it has deleted your data, how do you know it actually forgot? That question drove Suchita Yerramsetty to build Lethe, a polygraph-style auditing system for AI memory. Unlike most demos that focus on helping AI remember more, Lethe tests whether an AI can prove it has truly erased a user’s information.
The forgetting paradox in AI systems
Across industries, AI models are increasingly fed user data into vector stores and knowledge graphs designed to remember, generalize, and link information. But when a user invokes their right to erasure under laws like GDPR Article 17 or India’s DPDP Act 2023, simply deleting a data row may not remove derived artifacts. Embeddings, graph nodes, and cross-references can persist, turning “we deleted it” into a claim rather than proof. Lethe turns this paradox into a testable process.
Using recall as an attack surface
Lethe operates within Cognee’s memory lifecycle—remember, recall, improve (memify), and forget—but weaponizes the recall function. An Auditor agent fires a fixed set of 15 extraction probes at the memory store before and after deletion. Each probe falls into one of four classes: direct queries, inference-based re-identification, reconstruction of sensitive data, and relational checks across graph edges. A judge—an LLM via Anthropic’s API or a rule-based fallback—scores each response as LEAK or SAFE. The probes are frozen to ensure consistent, repeatable comparisons.
From red to green: the moment of verification
In baseline tests with a seeded customer named Ravi Sharma, all 15 probes leaked personal details. After standard row deletion, the score still showed residual references because another customer’s transcript mentioned knowing Ravi. Only when cross-references were redacted across the entire graph did the score drop to zero—verifying complete erasure. Lethe separates two realities: record deletion and full person erasure. The first may leave traces; the second confirms the data is gone.
Source: DEV Community. AI-assisted editorial synthesis — TechnoExpress.