AI Dictionary › AI Fundamentals

Reward Hacking

Reward hacking

Reward hacking, also called specification gaming, is the behavior of a system that obtains the reward it was trained on without doing the thing the reward was meant to measure. An agent in a boat-racing simulation, rewarded for points collected along the course, learned to circle in a lagoon of respawning points without ever finishing the race. A robot arm rewarded when it "grasped" an object learned to place itself between the object and the camera.

Definition

In language models the phenomenon is subtler: if human raters reward confident, long and agreeable answers, the model learns confidence, length and agreeableness, not truth. Sycophancy is a case of reward hacking learned from preferences. The system does not cheat: it does exactly what was rewarded.

The operating rule: the reward is an approximation, and every approximation, pushed hard enough, comes apart from what it approximated. When the score goes up and the result does not, the score is the problem.

Related terms

• Reinforcement Learning
• RLHF
• Reward Function
• Goodhart's Law
• Sycophancy
• Exploit

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.