AI Dictionary › AI Fundamentals

Gradient Accumulation

Accumulo del gradiente

Gradient accumulation is a technique that simulates a larger training batch than the available memory would directly allow. Instead of updating the weights after every small batch, the gradients computed over several consecutive batches are summed, and the weight update only happens after a preset number of batches has been accumulated. This achieves the effect of a large batch while using hardware with limited memory.

Definition

How it works

At each step, the model computes gradients on a small batch, but instead of applying them immediately it adds them to those from previous steps. Only after a certain number of accumulated steps does the optimizer apply the overall update to the weights, as if it had been computed on one larger batch.

Applications

It is particularly useful when training large-scale models on GPUs with limited memory, preserving the training stability typical of large batches without needing more expensive hardware. It is a common technique in distributed training and in fine-tuning large language models.

History & etymology

It is a practical engineering solution that emerged to work around hardware memory limits when training increasingly large neural networks.

Related terms

• Gradient Descent
• GPU and TPU
• Gradient Clipping
• Compute
• Learning Rate

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.