AI Dictionary › AI Fundamentals

Pre-training

Pre-addestramento

Pre-training is the first and most expensive phase of building a large model, where it learns from enormous amounts of raw, unlabeled data. For a language model this means reading billions of sentences and learning to predict the next word, thereby absorbing grammar, facts, styles, and reasoning patterns. It's like a person's general education: before specializing in a trade, they accumulate broad background knowledge by reading and observing the world. This phase produces the foundation model, a reservoir of general knowledge not yet aimed at a specific task.

Definition

Pre-training takes months of computation across thousands of processors and is what makes models so versatile. Only afterward, with lighter phases like fine-tuning, is this base adapted to specific uses, far more efficient than training from scratch for every application.

Related terms

• Foundation Model
• Fine-tuning
• Transfer Learning
• Training Data
• Scaling Laws

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.