AI Dictionary › AI Fundamentals

Data cleaning

Data cleaning is the set of operations used to identify and correct errors, inconsistencies, duplicates and missing values in a dataset before using it to train a machine learning model. It is often the least glamorous but most decisive phase of a data project: a sophisticated model trained on dirty data produces unreliable results, regardless of how advanced the chosen algorithm is.

Definition

How it works

The process includes several typical activities: identifying and handling missing values, removing or correcting duplicate records, standardizing inconsistent formats such as dates written in different ways, spotting clearly wrong or out-of-range values, such as an age of 200 years, and checking consistency between related fields. It often also requires standardizing units of measurement, correcting typos in text fields, and resolving ambiguities in category encoding, for example when the same city appears written in several different ways within the same dataset.

Applications

In applied AI, data cleaning almost always precedes feature engineering and is responsible, according to various industry estimates, for most of the time spent on a typical data science project, often more than the time devoted to choosing and training the actual model. It is especially critical in regulated sectors like healthcare and finance, where dirty data can translate into flawed automated decisions with direct consequences on real people.

History & etymology

The concept of checking and correcting data quality belongs to applied statistics and enterprise data processing well before machine learning, with roots in the quality control practices of censuses and statistical surveys throughout the twentieth century. With the exponential growth of automatically collected data volumes from the 1990s onward, data cleaning established itself as a distinct discipline within the broader field of data governance, with dedicated tools and methodologies.

Related terms

• Missing values
• Outlier
• Dataset
• Feature engineering
• Data Governance

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.