AI Dictionary › AI Fundamentals
Throughput is the amount of work a system completes per unit of time: requests per second, tokens per second, transactions per minute. It measures capacity, to be distinguished from latency, which measures the speed of a single operation. A ferry has high throughput and high latency; a speedboat the opposite: it carries little, yet arrives fast.
Throughput and latency are often in tension: techniques like batching raise overall capacity at the price of making individual requests wait, while optimizing for instant response can waste capacity. Real throughput is set by the bottleneck: CPU, memory, network bandwidth or disk I/O. Finding and widening it is the craft of performance optimization.
In AI, throughput is measured in tokens per second: inference systems like vLLM use continuous batching to serve many requests together and multiply tokens generated per GPU, slashing unit cost. In training, what matters is examples processed per second across the whole cluster. Beyond AI it drives the sizing of databases, networks and payment systems.
The English word combines through and put: what passes through the system. It was already used in manufacturing and railways for productive or transport capacity; computing and telecommunications adopted it for the capacity of channels and processors, and Erlang's queueing theory has provided the mathematical tools for analyzing it since 1909.
Grace's 7 evaluation dimensions measure both the quality and the efficiency of your prompts: thinking in terms of throughput helps you produce more with the same number of tokens. In the scenarios on AI costs and performance, distinguishing throughput from latency is the expert move.
From our network
AGORÀ Intelligence: Enterprise AI Governance Platform
Govern AI at scale: policies, adoption and measurable results on your data. Built for boards and C-suite.
Visit agora-intelligence.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.