AI Dictionary › AI Fundamentals
Teacher forcing is a training technique for sequential models, such as language models, in which during training the model is given the correct word from the real sequence as input, instead of its own prediction from the previous step. It makes training more stable and efficient, preventing small errors from accumulating along the sequence during the learning phase.
When generating text, a model produces output one element at a time, and would normally use its own previous prediction as input for the next step. With teacher forcing, during training this input is replaced with the correct value taken from the real data, so that the model's occasional errors do not propagate and harm the learning of subsequent steps.
It is common practice when training sequence-to-sequence models, including encoder-decoder architectures and transformer-based language models. It is worth noting that at inference time the model has no access to real correct values and must rely on its own predictions, a difference between training and use that developers need to account for.
It is an established technique in the training of recurrent neural networks and sequential models, adopted to speed up convergence during supervised training.
From our network
HSE Genius: AI for Safety Data Sheets
Extract SDS data, H phrases and ECHA compliance checks in seconds, powered by AI.
Visit hsegenius.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.