AI Dictionary › AI Fundamentals

Attention Mechanism

The Attention Mechanism is the key component of the Transformer architecture that allows the model to "pay attention" to the most relevant parts of the text when generating each token of the output. In practice, to produce each word of the response, the model calculates how relevant every other word in the input is for that specific position.

Definition

Attention is what allows LLMs to grasp long-range semantic relationships in text.

Related terms

• Transformer
• Neural Network
• LLM (Large Language Model)
• Context Window
• Embedding

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.