AI Dictionary › AI Models
The KV cache is an optimization technique used during text generation with attention-based models, which consists of saving and reusing the key and value vectors computed for already-processed tokens, instead of recomputing them every time a new token is generated. It drastically reduces the amount of repeated computation needed during sequential text generation.
When a model generates text one token at a time, computing the new token requires relating, through attention, its representation to that of every preceding token in the sequence. Without caching, this would require recomputing from scratch the keys and values of all previously generated tokens for every single new token produced. With the KV cache, these vectors are computed only once per token and kept in memory, allowing them to be reused directly in subsequent steps.
It is an essential component of every efficient inference system for large language models: it largely determines the amount of memory needed to serve requests with long contexts and directly affects the latency and throughput of text generation systems. Techniques such as sharing keys and values across multiple attention heads were developed specifically to reduce the KV cache's memory footprint in more recent models.
The name comes directly from the elements being cached, the keys and values of the attention mechanism, and it is a technique that became central with the large-scale spread of generative language models based on attention architectures.
From our network
Kaimaki Web: Websites That Win Customers
Custom websites, web apps and digital marketing for growing businesses.
Visit kaimakiweb.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.