AI Dictionary › AI Fundamentals

Attention Mechanism

The Attention Mechanism is the key component of the Transformer architecture that allows the model to "pay attention" to the most relevant parts of the text when generating each token of the output. In practice, to produce each word of the response, the model calculates how relevant every other word in the input is for that specific position.

Definition

Attention is what allows LLMs to grasp long-range semantic relationships in text: the word "it" on page 3 of a document can be linked back to its antecedent "the CEO" on page 1, even with dozens of paragraphs in between. This ability underlies modern models' contextual understanding.

Extending attention to multiple parallel "heads" (Multi-Head Attention) lets the model grasp several kinds of relationships in the text at once, grammatical, semantic, anaphoric, contributing to the richness and coherence of the generated output. For non-technical professionals, the key intuition is that attention is why modern models "understand" context in a surprisingly human way.

Related terms

• Transformer
• Neural Network
• LLM (Large Language Model)
• Context Window
• Embedding

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.