AI Dictionary › AI Models
Self-attention is the mechanism by which each element of a sequence (for example, each token of a text) computes its own representation by comparing itself with every other element in the same sequence, including itself. Unlike attention that relates two different sequences, here the input and the reference coincide: the model learns which words in the same sentence are relevant to interpreting a given word.
Technically, each token is projected into three vectors called query, key and value. The query of the current token is compared against the keys of every token in the sequence to produce relevance scores, which are normalized and used to compute a weighted sum of the values. The result is a new representation of the token that incorporates contextual information from across the whole sequence, regardless of the distance between words.
It is the central building block of architectures that process text, code and other sequences in parallel rather than strictly step by step: it captures long-range dependencies, such as the link between a pronoun and the noun it refers to several sentences earlier, without having to traverse the sequence step by step as recurrent models did.
The term spread from the second half of the 2010s within research on sequence-to-sequence architectures based exclusively on attention mechanisms, in contrast to the recurrent and convolutional models that had dominated natural language processing until then.
From our network
AGORÀ Intelligence: Enterprise AI Governance Platform
Govern AI at scale: policies, adoption and measurable results on your data. Built for boards and C-suite.
Visit agora-intelligence.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.