AI Dictionary › AI Fundamentals

Training Data Extraction

Estrazione dei Dati di Addestramento

Training data extraction is a technique by which an attacker induces a language model to reproduce verbatim fragments of text memorized during training, rather than generating new, generalized text. It is the most direct risk tied to unintentional memorization: if a model has seen a certain document enough times, or if that document was particularly distinctive, it may repeat it almost identically when given a prompt that recalls its beginning.

Definition

How it works

Technically the attack often works by feeding the model an initial fragment of a text suspected to be in the training set and observing whether the model completes the rest too precisely to be coincidental, or by repeating the same request many times with small variations to increase the chance of surfacing a memorized fragment. Larger models, paradoxically, tend to memorize rare or distinctive text passages more easily than smaller ones.

Applications

It is a concrete risk when training data includes personal information, proprietary code, or copyrighted material: a model could return, verbatim, data that should never appear in output, such as phone numbers, email addresses, or entire passages from copyrighted books. Main defenses include deduplicating training data, which reduces the repetitions that favor memorization, and differential privacy techniques that mathematically limit how much a single document can imprint itself on the model.

History & etymology

The phenomenon was publicly and systematically documented by the research community starting in 2020-2021, when experiments on large language models showed it was possible to extract verbatim memorized text with relatively simple requests, prompting AI labs to invest in specific mitigations.

Related terms

• Model Inversion Attack
• Membership Inference Attack
• Training Data
• Copyright and AI
• Overfitting

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.