AI Dictionary › Fondamenti AI

Teacher Forcing

Teacher forcing is a training technique for sequential models, such as language models, in which during training the model is given the correct word from the real sequence as input, instead of its own prediction from the previous step. It makes training more stable and efficient, preventing small errors from accumulating along the sequence during the learning phase.

Definition

What it is

Teacher forcing is a training technique for sequential models, such as language models, in which during training the model is given the correct word from the real sequence as input, instead of its own prediction from the previous step. It makes training more stable and efficient, preventing small errors from accumulating along the sequence during the learning phase.

How it works

When generating text, a model produces output one element at a time, and would normally use its own previous prediction as input for the next step. With teacher forcing, during training this input is replaced with the correct value taken from the real data, so that the model's occasional errors do not propagate and harm the learning of subsequent steps.

Applications

It is common practice when training sequence-to-sequence models, including encoder-decoder architectures and transformer-based language models. It is worth noting that at inference time the model has no access to real correct values and must rely on its own predictions, a difference between training and use that developers need to account for.

History & etymology

It is an established technique in the training of recurrent neural networks and sequential models, adopted to speed up convergence during supervised training.

Definizione (italiano)

Il teacher forcing è una tecnica di addestramento per modelli sequenziali, come i modelli linguistici, in cui durante il training il modello riceve come input la parola corretta della sequenza reale invece della propria previsione del passo precedente. Serve a rendere l'addestramento più stabile ed efficiente, evitando che piccoli errori si accumulino lungo la sequenza durante la fase di apprendimento.

Nella generazione di testo, un modello produce un output un elemento alla volta, e normalmente userebbe la propria previsione precedente come input per il passo successivo. Con il teacher forcing, durante l'addestramento questo input viene sostituito con il valore corretto tratto dai dati reali, così che gli errori occasionali del modello non si propaghino e non compromettano l'apprendimento dei passi successivi.

È una pratica comune nell'addestramento di modelli sequenza-a-sequenza, incluse le architetture encoder-decoder e i modelli linguistici basati su transformer. Va notato che a inferenza il modello non ha accesso ai valori corretti reali e deve basarsi sulle proprie previsioni, una differenza tra addestramento e utilizzo di cui gli sviluppatori devono tenere conto.

È una tecnica consolidata nell'addestramento di reti neurali ricorrenti e modelli sequenziali, adottata per accelerare la convergenza durante il training supervisionato.

Related terms

• Encoder-Decoder
• Recurrent Neural Network (RNN)
• Supervised Learning
• Long Short-Term Memory (LSTM)
• Transformer

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario — apply this concept with an instant AI verdict.
• Daily gym — short drills to train dictionary concepts.
• Playoff game — challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified — the AI coach that trains and certifies your prompt engineering — by Agora Intelligence.