Embeddings leak too

A fingerprint of meaning

Imagine a librarian who, instead of filing books by title, gives each one a position on a huge map: cookbooks in one corner, travel guides in another, books about Italian cooking near both. Two books close together on the map are about similar things. An embedding is that position, for a piece of text.

A tiny made-up example with only three numbers per text (real ones have hundreds):

notes = {
    "my knee hurts after running":   [0.91, 0.10, 0.02],
    "booked physio for my knee":     [0.88, 0.15, 0.05],
    "new recipe for lentil soup":    [0.03, 0.12, 0.95],
}
# The two knee notes sit close together; the soup is far away.

Leak 1: the numbers can be turned back into words

Because embeddings are designed to capture meaning, anyone who has them can learn a lot. Researchers have shown that, with the right tools and access to the same model, much of the original wording of short texts can be reconstructed from their embeddings. Even without that, someone can compare your stored embeddings with embeddings of guesses ("is any note about a knee injury?") and see which are close.

Leak 2: membership inference

Sometimes the secret is not what a record says but whether it exists. If a clinic's AI search was built only from patients' notes, then learning "this person's note is in there" already reveals they are a patient.

Here is the idea made simple. Suppose a search tool returns a "similarity score" from 0 to 1. An attacker who already has a copy of a note (say, a leaked email) asks the tool to find it:

• If the note is in the database, the best match scores about 1.00, because it is matching itself.
• If the note is not there, the best match is merely similar, say 0.80.

That gap answers the question "is this person in here?". The same pattern appears with trained models: a model tends to be more confident on examples it was trained on than on new ones, and an attacker can measure that difference.

Defences

• Treat embeddings as personal data. Protect them like the text they came from: access control, encryption at rest, deletion when the text is deleted.
• Redact before you embed. The redactor from the last lesson works here too: remove names and contact details first.
• Do not show raw scores. Show results, or rounded bands like "good match", so nobody can spot the near-perfect self-match.
• Keep the index on the device when it is for one person's own notes. Nobody else can query it.
• Add careful noise when training on many people's data. This is called differential privacy; the deep half of the course covers it properly.

A common misconception

"We only store vectors, not text, so no personal data is kept." The vectors were made to preserve meaning. If the meaning was personal, the vectors are personal too.