AI Dictionary › AI Fundamentals

Latency

Latenza (Latency)

Latency is the time between sending a prompt and receiving the model's response. It is typically measured in two ways: time to first token (how long you wait before the response starts appearing) and total generation time.

Definition

It depends on several factors: model size (larger models are slower), prompt and response length, server load, and inference optimization techniques. That's why providers often offer multiple versions: a powerful but slower model for complex tasks, a fast one for real-time interactions.

In AI product design, latency is a decisive UX factor: a customer service chatbot must answer in 1-2 seconds, document analysis can afford 30. Prompt efficiency helps too: denser prompts generate faster responses.

Related terms

• Inference
• Token
• SLM (Small Language Model)
• API
• Token Efficiency

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.