Course Outline – beginner friendly, hands-on, and practical
⏳ ~6 weeks 🧠 no prior coding required 📊 real datasets
Data cleaning is one of the most important steps in any data project. This course will teach you how to use Python to find, fix, and prevent common data problems – missing values, duplicates, inconsistent formats, outliers, and more. By the end, you will be able to prepare raw data for analysis, dashboards, or machine learning.
🎯 Goal: write clean Python code to clean, reshape, and validate real-world data (CSV, Excel, JSON).
pandas? Series and DataFramesisnull(), info())dropna(), fillna())to_datetime(), timezones)duplicated(), drop_duplicates())groupby()pipe()df.info())📌 Next steps
Welcome to the "Python for Data Cleaning" course! This is Module One. In this module, we will start from the very beginning. We will learn what data cleaning is, why it is important, and how Python can help us. We will also set up our computer to start writing Python code. By the end of this module, you will understand the basics of data cleaning and be ready to start coding. Let's begin our journey!
After this module, you will be able to:
Chidi wanted to buy ingredients for a big family dinner. He wrote a list on a piece of paper. But the list was messy. Some items were written twice. Some had wrong spellings. Some numbers were missing. Chidi couldn't read his own list properly!
His sister said: "Let's clean this list." They fixed the spellings, removed duplicates, and added the missing numbers. Now the list was perfect. They went to the market and bought everything they needed.
Data cleaning is just like that. We take messy data and make it clean and organized. Python helps us do this quickly and easily. Let's learn how!
Definition: Data is information. It can be numbers, words, pictures, or sounds.
Why it is important: Data helps us understand the world. We use data to make decisions.
Simple explanation: Think of data as pieces of a puzzle. Alone, they don't make sense. Together, they show a picture.
Real-life example: Your school grades are data.
School example: A teacher records attendance. That is data.
Home example: Your family's shopping list is data.
Nigerian example: A farmer's crop yield records are data.
Illustration:
Data Pieces:
+---------+ +---------+ +---------+
| Name | | Age | | City |
+---------+ +---------+ +---------+
| Chidi | | 12 | | Lagos |
| Ama | | 14 | | Abuja |
| Bola | | 13 | | Ibadan |
+---------+ +---------+ +---------+
Mini summary: Data is information that helps us understand things.
Definition: Data cleaning is the process of finding and fixing errors in data.
Why it is important: Dirty data can lead to wrong decisions. Clean data is reliable.
Simple explanation: It's like cleaning your room. You remove trash and organize things so you can find what you need.
Real-life example: A company cleans its customer list so that it can send emails to the right people.
School example: A teacher cleans the gradebook to make sure all scores are correct.
Home example: You organize your school bag so you don't forget anything.
Nigerian example: A bank cleans its transaction records to find errors.
Illustration:
Dirty Data -> Clean Data
+----------------------+ +----------------------+
| Chidi, 12, Lagos | | Chidi, 12, Lagos |
| Ama, 14, Abuja | | Ama, 14, Abuja |
| Bola, 13, Ibadan | | Bola, 13, Ibadan |
| Chidi, 12, Lagos | --> | (duplicate removed) |
| Ama, 14, AbuJa | | Ama, 14, Abuja |
+----------------------+ +----------------------+
Mini summary: Data cleaning means fixing errors in data.
Definition: Data cleaning is important because clean data leads to accurate analysis and good decisions.
Why it is important: Dirty data can cost time, money, and trust.
Simple explanation: If you use a dirty map, you might get lost. Clean data is like a clean map—it shows you the right way.
Real-life example: A hospital uses clean patient data to give the right treatment.
School example: A school uses clean attendance data to know which students need help.
Home example: A family uses a clean budget to manage money.
Nigerian example: A Nigerian government uses clean census data to plan for schools and hospitals.
Illustration:
Dirty Data -> Bad Decisions -> Problems
Clean Data -> Good Decisions -> Success
Mini summary: Clean data helps us make good decisions.
Definition: Common data problems include missing values, duplicates, wrong spellings, and inconsistent formats.
Why it is important: Knowing the problems helps us fix them.
Simple explanation: These are like the "dirt" in dirty data.
Real-life example: A customer's phone number is missing. That is a missing value.
School example: A student's name is spelled differently in two records. That is inconsistency.
Home example: You wrote the same item twice on your shopping list. That is a duplicate.
Nigerian example: A farmer records crop yield in different units (kg and tons). That is inconsistency.
Illustration:
Common Problems:
Missing values: ___
Duplicates: Chidi, Chidi
Wrong spellings: AbuJa, Abuja
Inconsistent: 12 kg, 0.012 tons
Mini summary: Dirty data has problems like missing values, duplicates, and inconsistencies.
Definition: Python is a programming language that is easy to learn and powerful.
Why it is important: Python is used by data analysts, scientists, and developers to clean data.
Simple explanation: Python is like a tool that helps you work with data. You give it instructions, and it does the work.
Real-life example: Many companies use Python to clean their data.
School example: Students use Python to analyze data for projects.
Home example: You can use Python to organize your family budget.
Nigerian example: Nigerian startups use Python to build apps and clean data.
Illustration:
Python Code -> Computer -> Clean Data
Mini summary: Python is a programming language for working with data.
Definition: Setting up Python means installing it on your computer so you can run code.
Why it is important: You need Python installed to start cleaning data.
Simple explanation: It's like installing a new app on your phone. You need it to use it.
Real-life example: You install Python from the official website.
School example: Your school computer lab may already have Python installed.
Home example: You can install Python on your personal computer.
Nigerian example: Many Nigerian universities have Python installed in their computer labs.
Illustration:
Step 1: Download Python.
Step 2: Install it on your computer.
Step 3: Open a code editor.
Step 4: Write your code.
Mini summary: Install Python to start coding.
Definition: Your first Python code is a simple command that tells the computer to do something.
Why it is important: It shows that Python is working and that you can give instructions.
Simple explanation: You write: print("Hello, World!") and the computer prints that message.
Real-life example: A programmer always starts with "Hello, World!" when learning a new language.
School example: You can print your name using Python.
Home example: You can use Python to print a message for your family.
Nigerian example: You can print "Hello, Nigeria!" with Python.
Illustration:
Code: print("Hello, World!")
Output: Hello, World!
Mini summary: Your first Python code prints a message.
Definition: A variable is a container that holds data, like a box with a label.
Why it is important: Variables let us store and work with data.
Simple explanation: Think of a variable as a labeled jar. You put data in it and use the label to find it.
Real-life example: age = 12, name = "Chidi"
School example: score = 85
Home example: budget = 5000
Nigerian example: population = 200000000
Illustration:
Variable: name = "Chidi"
Variable: age = 12
Mini summary: Variables store data for later use.
Definition: A list is a collection of items, like a shopping list.
Why it is important: Lists help us store multiple pieces of data together.
Simple explanation: You write: fruits = ["apple", "banana", "orange"]
Real-life example: A list of customer names.
School example: A list of student scores.
Home example: A list of groceries.
Nigerian example: A list of states in Nigeria.
Illustration:
fruits = ["apple", "banana", "orange"]
Mini summary: Lists store multiple items.
Definition: A dictionary stores data in key-value pairs. The key is like a label, and the value is the data.
Why it is important: Dictionaries help us organize data with labels.
Simple explanation: You write: person = {"name": "Chidi", "age": 12}
Real-life example: A customer record with name, phone, and address.
School example: A student record with name, grade, and subject.
Home example: A family member record with name and birthday.
Nigerian example: A city record with name and population.
Illustration:
person = {"name": "Chidi", "age": 12}
Mini summary: Dictionaries store data with labels.
Definition: Reading data from a file means opening a file and loading its content into Python.
Why it is important: Data is often stored in files (like CSV or Excel).
Simple explanation: You use Python to open a file and read its contents.
Real-life example: A company reads sales data from a CSV file.
School example: A teacher reads student grades from a file.
Home example: You read a family budget from a spreadsheet.
Nigerian example: A farmer reads crop data from a CSV file.
Illustration:
File: data.csv
Python reads the file.
Data is loaded into Python.
Mini summary: Python can read data from files.
Definition: Exploring data means looking at it to understand its structure and content.
Why it is important: You need to know what your data looks like before you clean it.
Simple explanation: You look at the first few rows, check the column names, and see what types of data you have.
Real-life example: A data analyst uses .head() to view the first rows.
School example: A student uses .info() to see data types.
Home example: You look at a receipt to see what you bought.
Nigerian example: A bank looks at transaction data to understand patterns.
Illustration:
data.head() -> Shows first 5 rows.
data.info() -> Shows column names and types.
Mini summary: Explore data to understand it before cleaning.
Definition: Missing values are empty spots in your data where information is missing.
Why it is important: You must handle missing values before analysis.
Simple explanation: You use data.isnull() to find missing values.
Real-life example: A customer has no email address.
School example: A student did not submit an assignment.
Home example: A family member forgot to write down an expense.
Nigerian example: A farmer did not record rainfall for one day.
Illustration:
data.isnull().sum() -> Counts missing values.
Mini summary: Identify missing values to fix them.
Definition: Data cleaning overview means understanding the whole process before you start.
Why it is important: It gives you a roadmap for cleaning data.
Simple explanation: The steps are: Explore, Identify Problems, Fix Problems, and Validate.
Real-life example: A data scientist follows this process.
School example: A student follows these steps for a class project.
Home example: You follow these steps to organize your receipts.
Nigerian example: A Nigerian business follows these steps to clean customer data.
Illustration:
Step 1: Explore data.
Step 2: Identify problems.
Step 3: Fix problems.
Step 4: Validate data.
Mini summary: Data cleaning is a step-by-step process.
Definition: Practice means applying what you learned to a real dataset.
Why it is important: Practice helps you learn and remember.
Simple explanation: You load a messy dataset and try to clean it.
Real-life example: A data analyst practices on a sample dataset.
School example: A student uses a sample dataset from the teacher.
Home example: You practice on a small list of expenses.
Nigerian example: A Nigerian student uses local data for practice.
Illustration:
Practice Dataset:
Name, Age, City
Chidi, 12, Lagos
Ama, 14, Abuja
Bola, 13, Ibadan
Chidi, , Lagos (missing age)
Ama, 14, AbuJa (wrong spelling)
Mini summary: Practice cleaning data.
How to Set Up Python in 4 Steps:
This module introduces the fundamental concepts of data cleaning and Python. Teachers should emphasize practical examples and encourage students to write code. Use interactive exercises and group activities to reinforce learning. Ensure students understand the importance of clean data before moving to more complex topics.
Parents can help children by encouraging them to think of data cleaning in daily life. Organizing a room, making a shopping list, or planning a schedule are all examples of data cleaning. Support children as they install Python and write their first code. Celebrate their small successes.
Data Cleaning Process
Dirty Data -> Explore -> Identify Problems -> Fix Problems -> Clean Data
Python Setup Flowchart
Download Python -> Install Python -> Install Code Editor -> Write Code
Dirty Data vs Clean Data
| Dirty Data | Clean Data |
|---|---|
| Has errors | No errors |
| Inconsistent | Consistent |
| Missing values | No missing values |
| Hard to use | Easy to use |
List vs Dictionary
| List | Dictionary |
|---|---|
| Ordered collection | Key-value pairs |
| Indexed by number | Indexed by key |
| fruits = ["apple", "banana"] | person = {"name": "Chidi"} |
In Module One, we learned what data and data cleaning are. We understood why data cleaning is important and identified common data problems. We introduced Python and set it up. We learned about variables, lists, and dictionaries. We practiced reading data, exploring it, and identifying missing values. We also learned the overall data cleaning process. These skills form the foundation for the rest of the course. You are now ready to start cleaning data with Python!
Match the term with its definition:
| Term | Definition |
|---|---|
| 1. Data | A. Fixing errors in data |
| 2. Data Cleaning | B. A programming language |
| 3. Python | C. A collection of items |
| 4. List | D. Information |
| 5. Dictionary | E. Key-value pairs |
Answers: 1-D, 2-A, 3-B, 4-C, 5-E
Scenario 1: You have a list of students' names with some missing values. What steps would you take to clean this data?
Scenario 2: A company has a customer list with duplicate entries. How would you clean it?
Scenario 3: A farmer records crop yields in different units (kg and tons). How would you make the data consistent?
In groups, create a list of 10 items with at least 3 errors (missing values, duplicates, wrong spellings). Exchange lists with another group and practice cleaning the data. Discuss the steps you took.
Write a short paragraph about a time you had to "clean" something—like organizing your room or fixing a list. Explain how it relates to data cleaning.
Create a "Data Cleaning Checklist" for yourself. Include steps like: explore data, check for missing values, remove duplicates, fix spellings, and validate. Use this checklist for future data cleaning tasks.
Create a small dataset (like a list of students with names, ages, and cities). Introduce 3 errors (missing values, duplicates, wrong spellings). Write Python code to load, explore, and clean the data.
Find a real dataset online (e.g., from Kaggle or a Nigerian data portal). Load it into Python and write a short report on its structure and any potential data cleaning issues you see.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Two, we will dive deeper into Python. We will learn about Pandas, a powerful library for data analysis. We will practice loading, exploring, and cleaning data more efficiently. Get ready to write more code and become a data cleaning expert!
End of Module One – You are now ready to start cleaning data with Python!
Welcome to Module Two! In the first module, we learned about data cleaning and the basics of Python. Now, we will learn about the most important tool for data cleaning in Python—Pandas. Pandas is like a superpower for working with data. It helps us load, explore, and clean data easily. In this module, we will focus on loading data and exploring it. By the end, you will be able to look at any dataset and understand its structure, types, and problems. Let's get started!
After this module, you will be able to:
A supermarket in Lagos wanted to understand its customers better. They had a huge file with all sales records. But the file was too big to read on paper. They needed a tool to explore the data quickly.
Their data analyst used Pandas. With just a few lines of code, she loaded the data, looked at the first few rows, checked for missing values, and summarized the sales. She found that many customers bought rice and beans together. She recommended a combo offer, and sales increased.
This module teaches you the skills that analyst used—loading data, exploring it, and understanding it. These are the first steps to cleaning and analyzing any dataset.
Definition: Pandas is a Python library that gives us tools to work with data.
Why it is important: Pandas makes data cleaning and analysis easy and fast.
Simple explanation: Think of Pandas as a magical toolbox for data. You put your data in, and you can do all kinds of things with it.
Real-life example: Data analysts use Pandas to clean and analyze data for companies.
School example: Students use Pandas to analyze survey data.
Home example: You can use Pandas to analyze your family's expenses.
Nigerian example: Nigerian businesses use Pandas to understand customer behavior.
Illustration:
Python + Pandas = Powerful Data Cleaning
Mini summary: Pandas is a Python library for working with data.
Definition: Installing Pandas means adding it to your Python environment.
Why it is important: You need Pandas installed to use it.
Simple explanation: It's like downloading a new app on your phone.
Real-life example: A data scientist installs Pandas before starting a project.
School example: A student installs Pandas in their computer lab.
Home example: You install Pandas on your personal computer.
Nigerian example: A Nigerian developer installs Pandas to build a data app.
Illustration:
pip install pandas
Mini summary: Install Pandas using pip install pandas.
Definition: Importing means telling Python that you want to use Pandas.
Why it is important: You must import Pandas before using it.
Simple explanation: It's like opening a toolbox before using a tool.
Real-life example: You write: import pandas as pd
School example: A student writes the import command in their notebook.
Home example: You write the import command in your code.
Nigerian example: A developer imports Pandas to start cleaning data.
Illustration:
import pandas as pd
Mini summary: Import Pandas with "import pandas as pd".
Definition: Loading data means reading a data file into Pandas.
Why it is important: You need to load data to work with it.
Simple explanation: You use pd.read_csv() to load a CSV file.
Real-life example: A data analyst loads sales data from a CSV file.
School example: A student loads a dataset for a class project.
Home example: You load your family's expense data from Excel.
Nigerian example: A Nigerian business loads customer data for analysis.
Illustration:
data = pd.read_csv("sales.csv")
Mini summary: Use pd.read_csv() to load CSV files.
Definition: head() is a function that shows the first few rows of data.
Why it is important: It gives you a quick look at the data.
Simple explanation: Like looking at the first page of a book.
Real-life example: An analyst uses head() to check the data format.
School example: A student uses head() to see the dataset.
Home example: You use head() to look at your expense data.
Nigerian example: A Nigerian business uses head() to inspect customer records.
Illustration:
data.head()
Mini summary: head() shows the first 5 rows of data.
Definition: tail() shows the last few rows of data.
Why it is important: It helps you see the end of the data.
Simple explanation: Like looking at the last page of a book.
Real-life example: An analyst checks the last few records.
School example: A student uses tail() to see the last rows.
Home example: You check the last entries in your expense log.
Nigerian example: A business checks the last transactions.
Illustration:
data.tail()
Mini summary: tail() shows the last 5 rows of data.
Definition: Data types are the types of data in each column, like numbers, text, or dates.
Why it is important: Knowing data types helps you clean data correctly.
Simple explanation: Numbers are int or float, text is object, dates are datetime.
Real-life example: An analyst checks data types before analysis.
School example: A student uses info() to check data types.
Home example: You check if your expense amounts are numbers.
Nigerian example: A business checks data types in customer records.
Illustration:
data.dtypes
Mini summary: Use dtypes to check data types.
Definition: info() gives a summary of the data, including column names, types, and non-null counts.
Why it is important: It gives a complete overview of the data.
Simple explanation: Like a summary sheet for your data.
Real-life example: An analyst uses info() to understand the dataset.
School example: A student uses info() to describe the dataset.
Home example: You use info() to check your family budget.
Nigerian example: A business uses info() to understand customer data.
Illustration:
data.info()
Mini summary: info() gives a summary of the data.
Definition: describe() gives statistics like mean, min, max, and count for numerical columns.
Why it is important: It helps you understand the distribution of numbers.
Simple explanation: Like a report card for your numbers.
Real-life example: An analyst uses describe() to see average sales.
School example: A student uses describe() to analyze test scores.
Home example: You use describe() to see average spending.
Nigerian example: A business uses describe() to understand transaction amounts.
Illustration:
data.describe()
Mini summary: describe() gives statistics for numerical data.
Definition: Missing values are empty or null values in the data.
Why it is important: Missing values can cause errors in analysis.
Simple explanation: You use isnull() and sum() to count missing values.
Real-life example: An analyst checks for missing customer emails.
School example: A student checks for missing assignment scores.
Home example: You check for missing expense entries.
Nigerian example: A business checks for missing customer data.
Illustration:
data.isnull().sum()
Mini summary: Use isnull().sum() to count missing values.
Definition: Duplicates are rows that appear more than once.
Why it is important: Duplicates can distort analysis.
Simple explanation: You use duplicated() and sum() to count duplicates.
Real-life example: An analyst checks for duplicate customer records.
School example: A student checks for duplicate survey responses.
Home example: You check for duplicate purchases.
Nigerian example: A business checks for duplicate transactions.
Illustration:
data.duplicated().sum()
Mini summary: Use duplicated().sum() to count duplicates.
Definition: value_counts() counts how many times each unique value appears in a column.
Why it is important: It helps you understand categorical data.
Simple explanation: Like counting how many students are in each grade.
Real-life example: An analyst counts the number of customers in each city.
School example: A student counts the number of students in each class.
Home example: You count how many times you bought each item.
Nigerian example: A business counts the number of customers in each state.
Illustration:
data['city'].value_counts()
Mini summary: value_counts() counts unique values in a column.
Definition: groupby() groups data by a column and allows you to perform calculations.
Why it is important: It helps you summarize data by categories.
Simple explanation: Like grouping students by class and finding the average score.
Real-life example: An analyst groups sales by city.
School example: A student groups test scores by subject.
Home example: You group expenses by category.
Nigerian example: A business groups sales by region.
Illustration:
data.groupby('city').mean()
Mini summary: groupby() groups data for summary analysis.
Definition: sort_values() sorts data by a column in ascending or descending order.
Why it is important: It helps you find the highest or lowest values.
Simple explanation: Like sorting test scores from highest to lowest.
Real-life example: An analyst sorts sales by amount.
School example: A student sorts grades to find the best and worst.
Home example: You sort expenses to see the largest amounts.
Nigerian example: A business sorts customers by purchase amount.
Illustration:
data.sort_values('sales', ascending=False)
Mini summary: sort_values() sorts data by a column.
Definition: A data exploration project is a complete process of loading, viewing, and summarizing a dataset.
Why it is important: It gives you hands-on practice with Pandas.
Simple explanation: You load a dataset, view it, check types, get statistics, and find missing values.
Real-life example: A data analyst explores a new dataset before cleaning.
School example: A student explores a class dataset.
Home example: You explore your family's expense data.
Nigerian example: A Nigerian business explores customer data.
Illustration:
data = pd.read_csv("file.csv")
data.head()
data.info()
data.describe()
data.isnull().sum()
data.duplicated().sum()
Mini summary: Practice exploring data with Pandas.
How to Explore Data with Pandas in 6 Steps:
This module introduces Pandas, the most important library for data cleaning in Python. Teachers should emphasize the importance of exploration before cleaning. Encourage students to practice with different datasets. Use hands-on exercises where students load, view, and summarize data. Highlight the practical use of Pandas in Nigeria and around the world.
Parents can help children by finding simple datasets to explore. For example, a list of monthly expenses or a list of books. Encourage children to practice loading and exploring data. Show them how data exploration is used in real life, like checking a budget or analyzing a shopping list.
Pandas Workflow
Load Data -> View Data -> Summarize Data -> Check Problems -> Clean Data
DataFrame Structure
+---------+---------+---------+
| Name | Age | City |
+---------+---------+---------+
| Chidi | 12 | Lagos |
| Ama | 14 | Abuja |
| Bola | 13 | Ibadan |
+---------+---------+---------+
head() vs tail()
| head() | tail() |
|---|---|
| Shows first rows | Shows last rows |
| Default 5 rows | Default 5 rows |
| Use to see top of data | Use to see bottom of data |
info() vs describe()
| info() | describe() |
|---|---|
| Shows column names, types, non-null counts | Shows statistics like mean, min, max |
| For all columns | For numerical columns only |
| Use to understand structure | Use to understand numbers |
In Module Two, we learned about Pandas, the most important Python library for data cleaning. We installed and imported Pandas. We learned how to load data from CSV files. We explored data using head(), tail(), info(), and describe(). We checked for missing values and duplicates. We also learned about value_counts(), groupby(), and sort_values(). These are the essential skills for exploring and understanding any dataset. You are now ready to start cleaning data with Pandas!
Match the function with its description:
| Function | Description |
|---|---|
| 1. head() | A. Shows last rows |
| 2. tail() | B. Gives summary |
| 3. info() | C. Shows first rows |
| 4. describe() | D. Counts unique values |
| 5. value_counts() | E. Gives statistics |
Answers: 1-C, 2-A, 3-B, 4-E, 5-D
Scenario 1: You have a CSV file of student grades. Load it and use head() to view the data. Then use info() and describe() to summarize it.
Scenario 2: You have a sales dataset. Check for missing values and duplicates. Count the number of missing values in each column.
Scenario 3: You have a customer dataset. Use value_counts() to see how many customers are from each city.
In groups, download a public dataset (e.g., from Kaggle or data.gov.ng). Load it into Pandas. Each group member takes turns exploring the data using head(), info(), describe(), and checking for missing values. Present your findings to the class.
Create a simple dataset (at least 10 rows and 3 columns) on a topic of your choice (e.g., friends, books, movies). Save it as a CSV file. Write Python code to load, explore, and summarize the data.
Create a "Data Exploration Report" for a dataset of your choice. Include: data loading, first and last rows, summary, data types, missing values, duplicates, and at least one groupby analysis. Present your report in a Jupyter Notebook.
Download a dataset from a Nigerian data portal (e.g., data.gov.ng). Load it into Pandas. Write a Python script that explores the data and prints: the first 5 rows, the last 5 rows, a summary, the number of missing values, and the number of duplicates.
Find a dataset with at least 5 columns. Use Pandas to explore it. Then, write a short report (in plain English) describing what the data is about, what you observed, and any potential problems you found (like missing values or inconsistencies).
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Three, we will learn how to handle missing values—one of the most common data cleaning tasks. We will practice filling, dropping, and imputing missing values. Get ready to clean your first dataset!
End of Module Two – You are now a Pandas explorer!
Welcome to Module Three! In the first two modules, we learned about data cleaning and how to explore data with Pandas. Now, we will tackle one of the most common problems in data cleaning—missing values. Missing values are like empty spaces in your data. They can cause problems if not handled properly. In this module, you will learn how to find, count, and fix missing values using Python. By the end, you will be able to clean datasets that have missing information. Let's get started!
After this module, you will be able to:
A school in Nigeria conducted a survey about students' favorite subjects. But some students didn't answer all the questions. Some left the age field blank. Others didn't say their favorite subject. The school principal wanted to know the top subject, but the missing answers made it hard.
A data analyst used Python to fix the problem. First, she found all the missing values. Then, she filled in the missing ages with the average age of the class. She dropped the rows where the favorite subject was missing because there were only a few. Now the data was clean, and she could see that Mathematics was the most popular subject.
This module will teach you how to handle missing values just like that analyst.
Definition: Missing values are empty or null values in a dataset. They are like blank spaces on a form.
Why it is important: Missing values can make your analysis wrong. You need to fix them before using the data.
Simple explanation: Imagine a class register where some students' names are missing. You can't know how many students are in the class.
Real-life example: A customer's phone number is missing in a database.
School example: A student forgot to write their age on a form.
Home example: You forgot to write down one expense in your budget.
Nigerian example: A farmer forgot to record rainfall on one day.
Illustration:
+---------+---------+
| Name | Age |
+---------+---------+
| Chidi | 12 |
| Ama | |
| Bola | 13 |
+---------+---------+
Mini summary: Missing values are empty spaces in data.
Definition: Missing values happen for many reasons—people forget, data isn't collected, or there are errors.
Why it is important: Understanding why data is missing helps you decide how to fix it.
Simple explanation: Sometimes people skip questions. Sometimes data is lost.
Real-life example: A customer didn't provide their email address.
School example: A student didn't answer a question on a survey.
Home example: You forgot to record a receipt.
Nigerian example: A bank may have missing data due to system errors.
Illustration:
Reasons for Missing Values:
1. People skip questions.
2. Data entry errors.
3. System failures.
4. Not applicable.
Mini summary: Missing values happen for various reasons.
Definition: You use Pandas functions like isnull() and notnull() to find missing values.
Why it is important: You need to know where the missing values are.
Simple explanation: isnull() shows True for missing values and False for non-missing.
Real-life example: A data analyst uses isnull() to find empty fields.
School example: A student uses isnull() to check for missing grades.
Home example: You use isnull() to find missing expenses.
Nigerian example: A business uses isnull() to find missing customer data.
Illustration:
data.isnull()
Mini summary: isnull() helps find missing values.
Definition: You can count missing values using isnull().sum().
Why it is important: Counting helps you understand how much data is missing.
Simple explanation: isnull().sum() counts the missing values in each column.
Real-life example: An analyst counts missing customer emails.
School example: A student counts missing test scores.
Home example: You count missing expenses.
Nigerian example: A business counts missing transaction data.
Illustration:
data.isnull().sum()
Mini summary: isnull().sum() counts missing values.
Definition: Missing values can cause errors in calculations and lead to wrong conclusions.
Why it is important: If you ignore missing values, your analysis might be wrong.
Simple explanation: If you calculate the average age and some ages are missing, the average will be wrong.
Real-life example: A company might target the wrong customers.
School example: A teacher might think students are doing well when they are not.
Home example: You might think you spent less money than you actually did.
Nigerian example: A government might plan for the wrong population.
Illustration:
Dirty Data -> Wrong Analysis -> Bad Decisions
Mini summary: Missing values lead to wrong conclusions.
Definition: You can either drop (remove) missing values or fill (replace) them with a value.
Why it is important: The choice depends on the data and the problem.
Simple explanation: If only a few rows are missing, you can drop them. If many rows are missing, you might want to fill them.
Real-life example: If a customer is missing an email, you might fill it with a default.
School example: If a student is missing a grade, you might fill it with the class average.
Home example: If you missed a receipt, you might estimate the amount.
Nigerian example: If a farmer missed a day of rainfall, you might fill it with an average.
Illustration:
Missing Values -> Drop or Fill?
Mini summary: Decide whether to drop or fill missing values.
Definition: dropna() removes rows or columns that have missing values.
Why it is important: It's a quick way to clean data when there are few missing values.
Simple explanation: If a row has any missing value, it is deleted.
Real-life example: A data analyst drops rows with missing customer emails.
School example: A student drops rows with missing grades.
Home example: You drop missing expenses from your budget.
Nigerian example: A business drops rows with missing transaction details.
Illustration:
data.dropna()
Mini summary: dropna() removes rows with missing values.
Definition: fillna() replaces missing values with a specified value.
Why it is important: It allows you to keep all the data and still fix missing values.
Simple explanation: You replace missing values with a number, a text, or an average.
Real-life example: You fill missing customer emails with "unknown@email.com".
School example: You fill missing grades with the class average.
Home example: You fill missing expenses with 0.
Nigerian example: You fill missing crop data with last year's average.
Illustration:
data.fillna(0)
Mini summary: fillna() replaces missing values with a value.
Definition: You can fill missing values with the mean (average), median (middle value), or mode (most common value).
Why it is important: These are logical ways to fill missing numbers.
Simple explanation: If you know the average age, you can fill missing ages with that average.
Real-life example: A company fills missing salaries with the median salary.
School example: You fill missing test scores with the class average.
Home example: You fill missing grocery costs with the average.
Nigerian example: A farmer fills missing rainfall data with the monthly average.
Illustration:
data.fillna(data.mean())
Mini summary: Use mean, median, or mode to fill missing values.
Definition: Forward fill (ffill) uses the previous value to fill a missing value. Backward fill (bfill) uses the next value.
Why it is important: Useful for time series data where the next or previous value makes sense.
Simple explanation: If Monday's data is missing, you use Sunday's data.
Real-life example: Stock prices are often filled using forward fill.
School example: If attendance is missing for one day, you use the previous day.
Home example: If you missed a daily expense, you use the previous day's expense.
Nigerian example: A farmer uses forward fill for missing daily temperature data.
Illustration:
data.fillna(method='ffill')
Mini summary: Forward and backward fill use neighboring values.
Definition: Interpolation estimates missing values based on the values around them.
Why it is important: It creates a smooth estimate for missing data.
Simple explanation: If you have points on a line, interpolation fills the missing point on the line.
Real-life example: Scientists use interpolation to fill missing climate data.
School example: You can interpolate missing scores based on previous and next scores.
Home example: You can interpolate missing temperatures.
Nigerian example: A farmer uses interpolation to estimate missing crop growth data.
Illustration:
data.interpolate()
Mini summary: Interpolation estimates missing values smoothly.
Definition: You drop when missing data is very little. You fill when missing data is significant.
Why it is important: The wrong choice can affect your analysis.
Simple explanation: If 1% of data is missing, drop it. If 20% is missing, fill it.
Real-life example: A company drops less than 5% missing data and fills more than that.
School example: A teacher drops a few missing grades but fills many.
Home example: You drop missing receipts that are few, but fill missing expenses that are many.
Nigerian example: A business drops a few missing records but fills many to keep the dataset.
Illustration:
Little Missing -> Drop
Lots Missing -> Fill
Mini summary: Drop when little is missing; fill when much is missing.
Definition: Categorical data is text data, like "city" or "gender".
Why it is important: You can't use mean for text. You can use mode or fill with "Unknown".
Simple explanation: If the city is missing, fill it with "Unknown" or the most common city.
Real-life example: A company fills missing customer city with "Unknown".
School example: A student fills missing subject with "Unknown".
Home example: You fill missing item category with "Other".
Nigerian example: A business fills missing region with "Unknown".
Illustration:
data['city'].fillna('Unknown')
Mini summary: Fill categorical missing values with "Unknown" or mode.
Definition: Practice applying all the techniques you learned.
Why it is important: Practice helps you remember and build skills.
Simple explanation: You take a messy dataset and fix all missing values.
Real-life example: A data analyst practices on a sample dataset.
School example: A student works on a class exercise.
Home example: You practice on your own expense data.
Nigerian example: A Nigerian student uses local data for practice.
Illustration:
data.dropna()
data.fillna(value)
data.fillna(data.mean())
data.fillna(method='ffill')
data.interpolate()
Mini summary: Practice all techniques to master missing values.
Definition: A project where you apply all missing value techniques to a real dataset.
Why it is important: It gives you real-world experience.
Simple explanation: You find a dataset, explore it, find missing values, and fix them.
Real-life example: A data analyst cleans a customer dataset.
School example: A student completes a class project.
Home example: You clean your family's expense data.
Nigerian example: A student uses a Nigerian dataset for the project.
Illustration:
Step 1: Load data.
Step 2: Explore data.
Step 3: Find missing values.
Step 4: Decide to drop or fill.
Step 5: Apply the chosen method.
Step 6: Validate the cleaned data.
Mini summary: Apply missing value techniques to a real dataset.
How to Handle Missing Values in 6 Steps:
This module is critical for data cleaning. Teachers should emphasize that handling missing values is a decision-making process. Encourage students to think about why data is missing and what method is best. Use real datasets with missing values. Have students practice dropping, filling with mean, forward fill, and interpolation. Discuss the pros and cons of each method.
Parents can help children understand missing values by using everyday examples. For instance, "If we don't know the price of one item, we can guess based on similar items." Encourage children to think about how they deal with missing information in daily life.
Missing Value Handling Flowchart
Missing Values -> Check -> Drop or Fill?
-> Drop: dropna() -> Clean Data
-> Fill: fillna() -> Clean Data
Forward Fill Example
Original: 10, 20, NaN, 40
Forward Fill: 10, 20, 20, 40
Drop vs Fill
| Drop | Fill |
|---|---|
| Removes rows. | Keeps rows. |
| Good when few missing. | Good when many missing. |
| Loses data. | Keeps data. |
Fill Methods
| Method | When to Use |
|---|---|
| Mean | Numerical data with normal distribution |
| Median | Numerical data with outliers |
| Mode | Categorical data |
| Forward Fill | Time series data |
| Interpolation | Smooth continuous data |
In Module Three, we learned how to handle missing values in Python using Pandas. We learned to find and count missing values with isnull().sum(). We explored two main approaches: dropping missing values with dropna() and filling them with fillna(). We filled using mean, median, mode, forward/backward fill, and interpolation. We also discussed how to handle categorical data and when to drop vs. fill. These skills are essential for cleaning real-world datasets. You are now ready to fix missing values in any dataset!
Match the term with its description:
| Term | Description |
|---|---|
| 1. isnull() | A. Removes missing values |
| 2. dropna() | B. Fills missing values |
| 3. fillna() | C. Finds missing values |
| 4. Mean | D. Most common value |
| 5. Mode | E. Average value |
Answers: 1-C, 2-A, 3-B, 4-E, 5-D
Scenario 1: You have a dataset of student grades. Some grades are missing. What would you do? Explain your steps.
Scenario 2: A company's customer data has missing emails for 60% of customers. What would you do?
Scenario 3: A farmer has daily rainfall data with some missing days. How would you fill the missing days?
In groups, get a dataset with missing values (from Kaggle or data.gov.ng). Each group member uses a different method to handle missing values (drop, fill with mean, forward fill, interpolation). Compare the results and discuss which method worked best.
Create a small dataset with missing values. Write Python code to handle the missing values using at least three different methods. Compare the results and write a short reflection on your choice.
Create a "Missing Values Report" for a dataset of your choice. Include: the number of missing values per column, your chosen method to handle them, and the cleaned data. Show before and after using head() and info().
Download a dataset with missing values (e.g., from data.gov.ng). Handle all missing values using appropriate methods. Write a report explaining your choices and show the cleaned dataset.
Find a dataset with missing values in both numerical and categorical columns. Use different methods for each type. Write a Python script that automates the handling of missing values.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Four, we will learn how to handle duplicate data, inconsistent formatting, and outliers. These are the next steps in cleaning messy data. Get ready to become a data cleaning expert!
End of Module Three – You are now a missing values master!
Welcome to Module Four! In the previous modules, we learned how to handle missing values. Now, we will tackle three more common data problems: duplicates, inconsistencies, and outliers. Duplicates are repeated rows. Inconsistencies are data that doesn't match—like "Lagos" and "lagos". Outliers are numbers that are very different from the rest—like an age of 150. By the end of this module, you will be able to find and fix all these problems. Let's begin!
After this module, you will be able to:
A business in Nigeria had a list of customers. But the list was messy. Some customers appeared twice. Some names were spelled differently—"Chidi" and "Chidi". Some ages were wrong—like 200 years old!
The business owner wanted to send a newsletter, but the duplicates meant some people would get it twice. The wrong ages made it hard to understand the customer base. A data analyst cleaned the list. She removed duplicates, fixed spellings, and removed the crazy ages. The list became clean and useful.
This module teaches you how to do the same.
Definition: Duplicates are rows that appear more than once in a dataset.
Why it is important: Duplicates can make your analysis wrong. They can count the same customer twice.
Simple explanation: Imagine a class register where a student's name appears twice. That would be a duplicate.
Real-life example: A customer is listed twice in a database.
School example: A student's name appears twice on a list.
Home example: You write the same item twice on a shopping list.
Nigerian example: A company has duplicate customer records.
Illustration:
+---------+---------+
| Name | Age |
+---------+---------+
| Chidi | 12 |
| Ama | 14 |
| Chidi | 12 | <- Duplicate
+---------+---------+
Mini summary: Duplicates are repeated rows.
Definition: You use duplicated() to find duplicate rows.
Why it is important: You need to know where the duplicates are.
Simple explanation: duplicated() returns True for duplicate rows.
Real-life example: A data analyst uses duplicated() to find repeated customer records.
School example: A student uses duplicated() to find repeated names.
Home example: You use duplicated() to find repeated expenses.
Nigerian example: A business uses duplicated() to find duplicate orders.
Illustration:
data.duplicated()
Mini summary: duplicated() finds duplicate rows.
Definition: You can count duplicates using duplicated().sum().
Why it is important: It tells you how many duplicates you have.
Simple explanation: duplicated().sum() counts all duplicate rows.
Real-life example: An analyst counts duplicate customer emails.
School example: A student counts duplicate names.
Home example: You count duplicate items.
Nigerian example: A business counts duplicate sales records.
Illustration:
data.duplicated().sum()
Mini summary: duplicated().sum() counts duplicates.
Definition: You use drop_duplicates() to remove duplicate rows.
Why it is important: It cleans the data by removing repeated rows.
Simple explanation: drop_duplicates() removes all duplicate rows and keeps only the first occurrence.
Real-life example: A company removes duplicate customers.
School example: A student removes duplicate names.
Home example: You remove duplicate items from your shopping list.
Nigerian example: A business removes duplicate orders.
Illustration:
data.drop_duplicates()
Mini summary: drop_duplicates() removes duplicates.
Definition: Inconsistencies are data that doesn't match. For example, "Lagos" and "lagos" are inconsistent.
Why it is important: Inconsistencies can cause errors in analysis.
Simple explanation: If some cities are written in uppercase and others in lowercase, they won't be counted as the same city.
Real-life example: A city is spelled "Abuja" and "abuja".
School example: A subject is written as "Math" and "math".
Home example: "Milk" and "milk" on a shopping list.
Nigerian example: "Lagos" and "lagos" in a customer database.
Illustration:
+---------+
| City |
+---------+
| Lagos |
| lagos |
| LAGOS |
+---------+
Mini summary: Inconsistencies are data that doesn't match.
Definition: Standardizing means making text data uniform. For example, converting all text to lowercase.
Why it is important: It ensures that "Lagos" and "lagos" are treated the same.
Simple explanation: You use str.lower() to convert text to lowercase.
Real-life example: A company standardizes city names.
School example: A student standardizes subject names.
Home example: You standardize your shopping list.
Nigerian example: A business standardizes customer city names.
Illustration:
data['city'] = data['city'].str.lower()
Mini summary: Standardize text using str.lower().
Definition: Whitespace is extra spaces before or after text.
Why it is important: Extra spaces can cause inconsistencies.
Simple explanation: You use str.strip() to remove leading and trailing spaces.
Real-life example: A company removes extra spaces in customer names.
School example: A student removes extra spaces in a list.
Home example: You remove extra spaces in your notes.
Nigerian example: A business removes spaces in product names.
Illustration:
data['name'] = data['name'].str.strip()
Mini summary: Use str.strip() to remove extra spaces.
Definition: You can use replace() to change values in a column.
Why it is important: It helps fix wrong spellings or inconsistent values.
Simple explanation: You replace "lagos" with "Lagos".
Real-life example: A company replaces "N/A" with "Unknown".
School example: A student replaces "Math" with "Mathematics".
Home example: You replace "milk" with "Milk".
Nigerian example: A business replaces "Abuja" with "Abuja" (capitalize).
Illustration:
data['city'] = data['city'].replace('lagos', 'Lagos')
Mini summary: Use replace() to fix values.
Definition: Pandas has many string methods to clean text data.
Why it is important: They make it easy to fix text data.
Simple explanation: Methods like str.lower(), str.strip(), str.replace().
Real-life example: A data analyst uses str methods to clean text.
School example: A student uses str methods to clean survey data.
Home example: You use str methods to clean a grocery list.
Nigerian example: A business uses str methods to clean product names.
Illustration:
data['name'] = data['name'].str.title()
Mini summary: Use str methods to clean text.
Definition: Outliers are values that are very different from the rest of the data.
Why it is important: Outliers can skew your analysis and lead to wrong conclusions.
Simple explanation: If most students are 12-14 years old, a student who is 150 years old is an outlier.
Real-life example: A salary of 1 billion naira is an outlier.
School example: A test score of 200 when the maximum is 100.
Home example: A grocery bill of 500,000 naira.
Nigerian example: A transaction amount of 10 million naira in a small business.
Illustration:
+---------+
| Age |
+---------+
| 12 |
| 13 |
| 14 |
| 150 | <- Outlier
+---------+
Mini summary: Outliers are extreme values.
Definition: IQR (Interquartile Range) is a method to find outliers. It looks at the middle 50% of data.
Why it is important: It's a standard way to identify outliers.
Simple explanation: You calculate Q1 (25th percentile) and Q3 (75th percentile). Then IQR = Q3 - Q1. Any value below Q1 - 1.5*IQR or above Q3 + 1.5*IQR is an outlier.
Real-life example: A data analyst uses IQR to find salary outliers.
School example: A student uses IQR to find grade outliers.
Home example: You use IQR to find spending outliers.
Nigerian example: A business uses IQR to find transaction outliers.
Illustration:
Q1 = data['age'].quantile(0.25)
Q3 = data['age'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5*IQR
upper = Q3 + 1.5*IQR
outliers = data[(data['age'] < lower) | (data['age'] > upper)]
Mini summary: IQR is a method to find outliers.
Definition: You can remove outliers or cap them to a maximum or minimum value.
Why it is important: Removing outliers can lose important data. Capping keeps the data but limits extreme values.
Simple explanation: You can drop outliers or replace them with the upper limit.
Real-life example: A company caps salaries at a certain limit.
School example: A teacher caps grades at 100.
Home example: You cap your spending at a limit.
Nigerian example: A bank caps transaction amounts.
Illustration:
Remove: data = data[(data['age'] >= lower) & (data['age'] <= upper)]
Cap: data['age'] = data['age'].clip(lower, upper)
Mini summary: Remove or cap outliers.
Definition: A boxplot is a chart that shows the distribution of data and highlights outliers.
Why it is important: It helps you see outliers visually.
Simple explanation: The boxplot shows the median, quartiles, and outliers as dots.
Real-life example: A data analyst uses a boxplot to find salary outliers.
School example: A student uses a boxplot to see grade distribution.
Home example: You use a boxplot to see spending distribution.
Nigerian example: A business uses a boxplot to see sales distribution.
Illustration:
import matplotlib.pyplot as plt
plt.boxplot(data['age'])
plt.show()
Mini summary: Boxplots help visualize outliers.
Definition: Practice applying all the techniques you learned.
Why it is important: Practice helps you remember and build skills.
Simple explanation: You take a messy dataset and clean all three types of problems.
Real-life example: A data analyst cleans a customer dataset.
School example: A student works on a class exercise.
Home example: You clean your own data.
Nigerian example: A Nigerian student uses local data.
Illustration:
Remove duplicates: data.drop_duplicates()
Standardize text: data['city'] = data['city'].str.lower().str.strip()
Remove outliers: using IQR method
Mini summary: Practice all three techniques.
Definition: A project where you apply all techniques to a real dataset.
Why it is important: It gives you real-world experience.
Simple explanation: You find a messy dataset and clean it completely.
Real-life example: A data analyst cleans a company dataset.
School example: A student completes a class project.
Home example: You clean your own expense data.
Nigerian example: A student uses a Nigerian dataset.
Illustration:
Step 1: Load data.
Step 2: Explore data.
Step 3: Remove duplicates.
Step 4: Fix inconsistencies.
Step 5: Handle outliers.
Step 6: Validate the cleaned data.
Mini summary: Apply all techniques to a real dataset.
How to Clean Duplicates, Inconsistencies, and Outliers in 6 Steps:
This module addresses three key data cleaning tasks. Teachers should emphasize the importance of each task with practical examples. Use datasets that contain duplicates, inconsistencies, and outliers. Show students how to use Pandas methods and the IQR technique. Encourage students to think about whether to remove or cap outliers.
Parents can help children understand these concepts by using everyday examples. For instance, "When we organize the pantry, we remove duplicate items and make sure all labels are consistent." Encourage children to think about how they clean and organize data in their daily lives.
Duplicate Removal Flowchart
Data -> Find Duplicates -> Remove Duplicates -> Clean Data
Boxplot Illustration
+---------------------+
| * Outlier |
| |
| +-------+ |
| | | |
| | Box | |
| | | |
| +-------+ |
| * Outlier |
+---------------------+
Remove vs Cap Outliers
| Remove | Cap |
|---|---|
| Deletes rows. | Keeps rows but limits values. |
| Good when few outliers. | Good when many outliers. |
| Loses data. | Keeps data. |
str.lower() vs str.strip()
| str.lower() | str.strip() |
|---|---|
| Converts to lowercase. | Removes leading/trailing spaces. |
| Fixes case inconsistency. | Fixes spacing inconsistency. |
In Module Four, we tackled three more data cleaning tasks: duplicates, inconsistencies, and outliers. We learned how to find and remove duplicates with duplicated() and drop_duplicates(). We fixed inconsistencies by standardizing text with str.lower(), str.strip(), and replace(). We identified outliers using the IQR method and decided whether to remove or cap them. We also visualized outliers using boxplots. These are essential skills for cleaning any real-world dataset. You are now ready to clean data like a professional!
Match the term with its description:
| Term | Description |
|---|---|
| 1. Duplicate | A. A repeated row |
| 2. Inconsistency | B. An extreme value |
| 3. Outlier | C. Data that doesn't match |
| 4. IQR | D. A method to find outliers |
| 5. Boxplot | E. A chart to visualize outliers |
Answers: 1-A, 2-C, 3-B, 4-D, 5-E
Scenario 1: You have a list of customers with duplicate records. What would you do?
Scenario 2: You have a city column with "Lagos", "lagos", and "LAGOS". How would you fix it?
Scenario 3: You have a salary column with values like 50,000, 60,000, and 10,000,000. How would you handle the outlier?
In groups, get a dataset with duplicates, inconsistencies, and outliers. Each group member takes one problem to fix. Combine your work to produce a clean dataset. Discuss the methods you used.
Create a dataset with at least 10 rows, including duplicates, inconsistencies, and outliers. Write Python code to clean all three types of problems. Show your cleaned dataset.
Create a "Data Cleaning Report" for a dataset of your choice. Include sections on duplicates, inconsistencies, and outliers. Show your methods and the cleaned data.
Download a messy dataset. Clean it by removing duplicates, fixing inconsistencies, and handling outliers. Write a report explaining your steps and show the cleaned dataset.
Find a dataset with all three problems. Write a Python script that automatically cleans the data. Include methods for duplicates, inconsistencies, and outliers. Test it on the dataset and show the results.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Five, we will learn about advanced data cleaning techniques like working with dates, merging datasets, and reshaping data. You will become a true data cleaning expert!
End of Module Four – You are now a data cleaning all-star!
Welcome to Module Five! In the previous modules, we cleaned missing values, duplicates, inconsistencies, and outliers. Now, we will learn three more important skills: working with dates, merging datasets, and reshaping data. Dates are often messy and need cleaning. Merging combines datasets. Reshaping changes how data is organized. By the end of this module, you will be able to handle almost any data cleaning task. Let's get started!
After this module, you will be able to:
A school in Nigeria had two separate databases. One had student names and IDs. The other had their test scores. They wanted to combine them to see each student's scores. They also had a column with dates in different formats—"12-05-2024" and "2024/05/12".
A data analyst used Pandas to merge the two datasets on student ID. She also converted all dates to a single format. Finally, she reshaped the data so that each subject had its own column. Now the school had a clean, unified dataset. This module teaches you these powerful skills.
Definition: Dates can be written in many formats—like "12-05-2024", "2024/05/12", or "May 12, 2024".
Why it is important: Inconsistent dates can cause errors in analysis.
Simple explanation: If some dates are in day-month-year and others are in month-day-year, they will be wrong.
Real-life example: A company has dates in different formats.
School example: A teacher records dates in different ways.
Home example: You write dates differently on different pages.
Nigerian example: A business has dates in "DD/MM/YYYY" and "MM/DD/YYYY".
Illustration:
12-05-2024 (DD-MM-YYYY)
2024/05/12 (YYYY/MM/DD)
May 12, 2024
Mini summary: Dates can be messy and need cleaning.
Definition: pd.to_datetime() converts a column to datetime format.
Why it is important: It standardizes dates and allows you to do date operations.
Simple explanation: You give Pandas a column of dates, and it converts them to a single, standard format.
Real-life example: A data analyst converts date strings to datetime.
School example: A teacher converts date formats.
Home example: You convert dates in your budget.
Nigerian example: A business converts dates to a standard format.
Illustration:
data['date'] = pd.to_datetime(data['date'])
Mini summary: pd.to_datetime() standardizes dates.
Definition: You can extract parts of a date like year, month, day, or day of the week.
Why it is important: It helps you analyze patterns over time.
Simple explanation: You can get the month from a date and see which month has the most sales.
Real-life example: A company analyzes sales by month.
School example: A teacher tracks attendance by day of the week.
Home example: You track expenses by month.
Nigerian example: A business analyzes sales by quarter.
Illustration:
data['year'] = data['date'].dt.year
data['month'] = data['date'].dt.month
data['day'] = data['date'].dt.day
Mini summary: Extract year, month, day from dates.
Definition: You can filter data based on dates, like selecting data from a specific year.
Why it is important: It helps you focus on a specific time period.
Simple explanation: You select rows where the date is in 2024.
Real-life example: A company analyzes sales for the last quarter.
School example: A teacher looks at grades for the second semester.
Home example: You look at expenses for a specific month.
Nigerian example: A business analyzes transactions in a specific year.
Illustration:
data_2024 = data[data['date'].dt.year == 2024]
Mini summary: Filter data by date.
Definition: Merging combines two datasets into one using a common column.
Why it is important: Often, data is stored in separate tables. Merging brings them together.
Simple explanation: You have a student list and a score list. You merge them on student ID.
Real-life example: A company merges customer data with sales data.
School example: A teacher merges student names with grades.
Home example: You merge your budget with your expenses.
Nigerian example: A business merges inventory data with sales data.
Illustration:
merged = pd.merge(data1, data2, on='key_column')
Mini summary: Merging combines datasets.
Definition: There are different types of merges: inner (only matching rows), outer (all rows), left (all from left), right (all from right).
Why it is important: You choose the type based on what data you want to keep.
Simple explanation: Inner merge keeps only rows that appear in both datasets.
Real-life example: You use inner merge to find customers who have made a purchase.
School example: You use left merge to keep all students, even those without scores.
Home example: You use outer merge to combine all data.
Nigerian example: A business uses left merge to keep all inventory items.
Illustration:
inner = pd.merge(data1, data2, on='key', how='inner')
left = pd.merge(data1, data2, on='key', how='left')
right = pd.merge(data1, data2, on='key', how='right')
outer = pd.merge(data1, data2, on='key', how='outer')
Mini summary: Choose the type of merge based on your needs.
Definition: Sometimes you need to merge on more than one column.
Why it is important: A single key may not be unique enough.
Simple explanation: You merge on both 'name' and 'city' to ensure a match.
Real-life example: A company merges on customer ID and location.
School example: A teacher merges on student name and class.
Home example: You merge on item name and store.
Nigerian example: A business merges on product code and region.
Illustration:
merged = pd.merge(data1, data2, on=['name', 'city'])
Mini summary: Merge on multiple columns.
Definition: Concatenating means stacking datasets on top of each other (rows) or side by side (columns).
Why it is important: You often need to combine data with the same columns.
Simple explanation: You have two datasets with the same columns. You stack them using pd.concat().
Real-life example: A company combines monthly sales data.
School example: A teacher combines grades from different classes.
Home example: You combine expenses from different months.
Nigerian example: A business combines branch sales data.
Illustration:
combined = pd.concat([data1, data2])
Mini summary: Concatenating stacks data.
Definition: Reshaping changes how data is organized—rows to columns or columns to rows.
Why it is important: Reshaping makes data easier to analyze.
Simple explanation: You have data where each subject is a row. You reshape to make each subject a column.
Real-life example: A company reshapes sales data for better reporting.
School example: A teacher reshapes test scores by subject.
Home example: You reshape your budget by category.
Nigerian example: A business reshapes sales by region.
Illustration:
Original: rows for each subject.
Reshaped: columns for each subject.
Mini summary: Reshaping changes the layout of data.
Definition: A pivot table summarizes data and allows you to view it in a new way.
Why it is important: It helps you analyze and summarize data quickly.
Simple explanation: You use pivot_table() to create a summary table.
Real-life example: A company creates a pivot table of sales by region.
School example: A teacher creates a pivot table of grades by subject.
Home example: You create a pivot table of expenses by category.
Nigerian example: A business creates a pivot table of sales by product.
Illustration:
pivot = data.pivot_table(index='city', columns='product', values='sales', aggfunc='sum')
Mini summary: Pivot tables summarize data.
Definition: Melting turns columns into rows. It is the reverse of pivoting.
Why it is important: Sometimes you need data in a long format for analysis.
Simple explanation: You use melt() to convert wide data to long data.
Real-life example: A company melts sales data for time series analysis.
School example: A teacher melts test scores for analysis.
Home example: You melt your budget for better tracking.
Nigerian example: A business melts sales data by quarter.
Illustration:
melted = data.melt(id_vars=['name'], value_vars=['math', 'science'])
Mini summary: Melting turns columns into rows.
Definition: Stacking compresses columns into rows. Unstacking does the opposite.
Why it is important: It gives you more control over the layout.
Simple explanation: You use stack() to make data longer, and unstack() to make it wider.
Real-life example: A company stacks data for analysis.
School example: A teacher stacks scores for each student.
Home example: You stack your expenses for tracking.
Nigerian example: A business stacks sales by product.
Illustration:
stacked = data.stack()
unstacked = data.unstack()
Mini summary: Stacking and unstacking change data layout.
Definition: groupby() groups data and allows you to apply functions like sum, mean, or count.
Why it is important: It's essential for summarizing data.
Simple explanation: You group by city and calculate the average sales.
Real-life example: A company groups sales by product.
School example: A teacher groups grades by class.
Home example: You group expenses by category.
Nigerian example: A business groups sales by region.
Illustration:
data.groupby('city')['sales'].mean()
Mini summary: groupby() summarizes data by category.
Definition: Practice applying all the techniques you learned.
Why it is important: Practice helps you remember and build skills.
Simple explanation: You take a messy dataset and clean dates, merge datasets, and reshape data.
Real-life example: A data analyst cleans a customer dataset.
School example: A student works on a class exercise.
Home example: You clean your own data.
Nigerian example: A Nigerian student uses local data.
Illustration:
Convert dates: pd.to_datetime()
Merge: pd.merge()
Reshape: pivot_table() or melt()
Mini summary: Practice all three techniques.
Definition: A project where you apply all advanced techniques to a real dataset.
Why it is important: It gives you real-world experience.
Simple explanation: You find a messy dataset with dates, multiple tables, and messy layout. You clean it completely.
Real-life example: A data analyst cleans a company dataset.
School example: A student completes a class project.
Home example: You clean your own expense data.
Nigerian example: A student uses a Nigerian dataset.
Illustration:
Step 1: Load data.
Step 2: Clean dates.
Step 3: Merge datasets.
Step 4: Reshape data.
Step 5: Validate the cleaned data.
Mini summary: Apply advanced techniques to a real dataset.
How to Clean Dates, Merge, and Reshape Data in 6 Steps:
This module covers advanced data cleaning topics. Teachers should emphasize the importance of dates and merging in real-world scenarios. Use examples with multiple datasets. Show students how to use pivot tables and melting effectively. Encourage practice with real datasets.
Parents can help children understand these concepts by using everyday examples like merging a shopping list with prices or converting dates on a calendar. Encourage children to think about how data is organized and how it can be reshaped.
Merging Flowchart
Data1 + Data2 -> Merge -> Combined Data
Pivot Table Illustration
Original:
City Product Sales
Lagos Rice 100
Abuja Rice 200
Lagos Beans 150
Abuja Beans 250
Pivot:
City Rice Beans
Lagos 100 150
Abuja 200 250
Merge Types
| Merge Type | Description |
|---|---|
| Inner | Keeps only matching rows |
| Outer | Keeps all rows |
| Left | Keeps all rows from left dataset |
| Right | Keeps all rows from right dataset |
Reshaping Methods
| Method | Action |
|---|---|
| pivot_table() | Summarizes data |
| melt() | Columns to rows |
| stack() | Compresses columns |
| unstack() | Expands rows |
In Module Five, we learned advanced data cleaning skills: working with dates, merging datasets, and reshaping data. We converted dates to datetime format, extracted date components, and filtered by date. We merged datasets using inner, outer, left, and right joins. We also learned how to concatenate data. For reshaping, we used pivot tables, melting, stacking, and unstacking. We also revisited groupby() for summary analysis. These skills are essential for handling complex real-world datasets. You are now a true data cleaning expert!
Match the term with its description:
| Term | Description |
|---|---|
| 1. to_datetime() | A. Combines datasets |
| 2. merge | B. Summarizes data |
| 3. pivot_table | C. Converts dates |
| 4. melt | D. Turns columns into rows |
| 5. groupby | E. Groups data for summary |
Answers: 1-C, 2-A, 3-B, 4-D, 5-E
Scenario 1: You have two datasets—student names and test scores. Merge them using student ID.
Scenario 2: You have a dataset with dates in multiple formats. Convert them to a single standard format.
Scenario 3: You have sales data with city, product, and sales. Create a pivot table showing sales by city and product.
In groups, get two related datasets. Merge them using the appropriate merge type. Then, reshape the merged data using a pivot table. Present your cleaned and reshaped data to the class.
Create a dataset with dates, merge it with another dataset, and reshape it using melt. Write Python code to perform all steps. Show your results.
Create a "Data Cleaning Report" for a dataset that includes dates, multiple tables, and messy layout. Clean the dates, merge the tables, and reshape the data. Show your methods and the final cleaned dataset.
Download a dataset with dates and merge it with another dataset. Clean the dates, merge the data, and reshape it. Write a report explaining your steps and show the final dataset.
Find a dataset with multiple tables and messy dates. Write a Python script that cleans the dates, merges the tables, and reshapes the data. Test it on the dataset and show the results.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Six, we will learn about automating data cleaning with functions and pipelines. You will learn how to write reusable code to clean any dataset. Get ready to become a data cleaning automation expert!
End of Module Five – You are now an advanced data cleaner!
Welcome to Module Six! In all the previous modules, we learned how to clean data step by step. But doing the same steps over and over for every dataset can be tiring. That's where automation comes in. We can write functions and create pipelines that automatically clean data for us. This saves time and reduces errors. By the end of this module, you will be able to write reusable code to clean any dataset. Let's begin!
After this module, you will be able to:
A company in Nigeria received new data every week. It always had the same problems—missing values, duplicates, and inconsistent text. A data analyst used to clean it manually every week. It took hours.
Then she wrote a Python function that did all the cleaning steps automatically. She ran the function on the new data each week. In minutes, the data was clean. She saved hours every week and never made mistakes.
This module teaches you how to write your own automated cleaning functions and pipelines.
Definition: Automation means using code to do repetitive tasks without human intervention.
Why it is important: It saves time, reduces errors, and makes your work consistent.
Simple explanation: Like a machine that washes dishes, automation does the work for you.
Real-life example: A company automates data cleaning for weekly reports.
School example: A student automates cleaning of class attendance data.
Home example: You automate sorting your expenses.
Nigerian example: A Nigerian business automates customer data cleaning.
Illustration:
Manual Cleaning: 2 hours
Automated Cleaning: 2 minutes
Mini summary: Automation saves time and reduces errors.
Definition: A function is a block of code that performs a specific task. You can reuse it whenever you need it.
Why it is important: Functions make your code organized and reusable.
Simple explanation: Like a recipe for making pancakes. You can use the same recipe every time.
Real-life example: A function that cleans missing values.
School example: A function that calculates average grades.
Home example: A function that organizes your to-do list.
Nigerian example: A function that standardizes Nigerian city names.
Illustration:
def clean_missing(data):
data = data.fillna(0)
return data
Mini summary: Functions are reusable blocks of code.
Definition: A cleaning function takes a dataset, applies cleaning steps, and returns the cleaned dataset.
Why it is important: It packages all your cleaning steps into one easy-to-use tool.
Simple explanation: You write a function that does all the cleaning steps you learned in previous modules.
Real-life example: A function that removes duplicates and fills missing values.
School example: A function that cleans student grades.
Home example: A function that cleans your budget.
Nigerian example: A function that cleans sales data.
Illustration:
def clean_data(data):
data = data.drop_duplicates()
data = data.fillna(0)
return data
Mini summary: A cleaning function applies all cleaning steps.
Definition: You can add multiple cleaning steps to your function.
Why it is important: One function can do all the cleaning work.
Simple explanation: You add steps for duplicates, missing values, text standardization, and outliers.
Real-life example: A function that removes duplicates, fills missing values, and standardizes text.
School example: A function that cleans attendance data.
Home example: A function that cleans your expense data.
Nigerian example: A function that cleans customer data.
Illustration:
def clean_data(data):
data = data.drop_duplicates()
data = data.fillna(0)
data['name'] = data['name'].str.lower().str.strip()
return data
Mini summary: Add all cleaning steps to one function.
Definition: Parameters are variables you pass to a function to make it more flexible.
Why it is important: You can customize the cleaning for different datasets.
Simple explanation: You can tell the function what value to fill missing data with.
Real-life example: A function that lets you choose the fill value.
School example: A function that lets you choose the columns to clean.
Home example: A function that lets you choose the date format.
Nigerian example: A function that lets you choose which city to standardize.
Illustration:
def clean_data(data, fill_value=0):
data = data.drop_duplicates()
data = data.fillna(fill_value)
return data
Mini summary: Parameters make functions flexible.
Definition: A pipeline is a sequence of steps that data goes through. In Pandas, you can use the pipe() function to create pipelines.
Why it is important: Pipelines make your code clean and organized.
Simple explanation: Think of a pipeline as a conveyor belt. Data goes in one end, and clean data comes out the other.
Real-life example: A data pipeline that loads, cleans, and saves data.
School example: A pipeline that cleans and analyzes student data.
Home example: A pipeline that cleans and sorts your expenses.
Nigerian example: A pipeline that processes sales data.
Illustration:
Data -> Step 1 -> Step 2 -> Step 3 -> Clean Data
Mini summary: Pipelines organize data cleaning steps.
Definition: The pipe() function allows you to chain multiple functions together.
Why it is important: It makes your code easier to read and maintain.
Simple explanation: You use pipe() to apply one function after another.
Real-life example: A pipeline that applies cleaning functions in order.
School example: A pipeline that cleans grades, then calculates averages.
Home example: A pipeline that cleans and sorts expenses.
Nigerian example: A pipeline that cleans and summarizes sales.
Illustration:
data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
Mini summary: pipe() chains cleaning functions.
Definition: Pipeline functions are functions designed to be used in a pipe. They take data as input and return cleaned data.
Why it is important: They are modular and reusable.
Simple explanation: Each function does one cleaning task. You can put them together in any order.
Real-life example: A function that drops duplicates, a function that fills missing values.
School example: A function that standardizes names.
Home example: A function that removes outliers.
Nigerian example: A function that standardizes city names.
Illustration:
def remove_duplicates(data):
return data.drop_duplicates()
def fill_missing(data):
return data.fillna(0)
def standardize_text(data):
data['name'] = data['name'].str.lower().str.strip()
return data
Mini summary: Pipeline functions are modular cleaning steps.
Definition: A complete cleaning pipeline combines all functions in the right order.
Why it is important: It automatically cleans data from start to finish.
Simple explanation: You chain all your cleaning functions using pipe().
Real-life example: A full pipeline for cleaning customer data.
School example: A full pipeline for cleaning attendance data.
Home example: A full pipeline for cleaning expenses.
Nigerian example: A full pipeline for cleaning sales data.
Illustration:
pipeline = data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
clean_data = pipeline
Mini summary: A complete pipeline cleans data automatically.
Definition: Testing means checking if your pipeline works correctly.
Why it is important: You need to make sure your pipeline cleans data properly.
Simple explanation: You run your pipeline on a small dataset and check the output.
Real-life example: A data analyst tests a pipeline on sample data.
School example: A student tests a pipeline on a small dataset.
Home example: You test your pipeline on a few days of expenses.
Nigerian example: A business tests a pipeline on a sample sales file.
Illustration:
test_data = pd.read_csv("test.csv")
cleaned = test_data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
print(cleaned.head())
Mini summary: Test your pipeline to ensure it works.
Definition: You can save your pipeline as a Python script or function to reuse it.
Why it is important: You can use the same pipeline on new datasets.
Simple explanation: You write your pipeline in a file and call it whenever you need to clean data.
Real-life example: A company saves a cleaning pipeline for weekly reports.
School example: A student saves a pipeline for class projects.
Home example: You save a pipeline to clean your monthly budget.
Nigerian example: A business saves a pipeline for quarterly sales data.
Illustration:
def cleaning_pipeline(data):
return data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
Mini summary: Save pipelines to reuse them.
Definition: Error handling means managing errors that might occur during cleaning.
Why it is important: Sometimes data is unexpected. You need to handle errors gracefully.
Simple explanation: You use try-except blocks to catch errors and continue.
Real-life example: A pipeline that handles missing columns.
School example: A pipeline that handles missing data.
Home example: A pipeline that handles missing files.
Nigerian example: A pipeline that handles missing fields.
Illustration:
def safe_clean(data):
try:
return data.pipe(remove_duplicates)
except Exception as e:
print("Error:", e)
return data
Mini summary: Handle errors to make pipelines robust.
Definition: Scheduling means running your pipeline automatically at set times.
Why it is important: It ensures data is cleaned regularly without manual effort.
Simple explanation: You use a scheduler to run your Python script every day or week.
Real-life example: A company schedules a pipeline to run every Sunday night.
School example: A teacher schedules a pipeline to run after each test.
Home example: You schedule a pipeline to run every month.
Nigerian example: A business schedules a pipeline for monthly reports.
Illustration:
# Example: Using a scheduler (like cron or Task Scheduler)
# Not code, just concept.
Mini summary: Schedule pipelines for regular cleaning.
Definition: Practice building a pipeline from scratch.
Why it is important: Practice makes you comfortable with automation.
Simple explanation: You take a messy dataset and build a pipeline to clean it.
Real-life example: A data analyst builds a pipeline for a new client.
School example: A student builds a pipeline for a class project.
Home example: You build a pipeline for your personal budget.
Nigerian example: A Nigerian student builds a pipeline for local data.
Illustration:
Step 1: Define cleaning functions.
Step 2: Chain them with pipe().
Step 3: Test on a dataset.
Step 4: Use on new data.
Mini summary: Practice building pipelines.
Definition: An automation project where you build a pipeline for a real dataset.
Why it is important: It gives you real-world experience.
Simple explanation: You find a messy dataset and build an automated cleaning pipeline.
Real-life example: A data analyst builds a pipeline for a company.
School example: A student completes a class project.
Home example: You build a pipeline for your own data.
Nigerian example: A student uses a Nigerian dataset.
Illustration:
Step 1: Load data.
Step 2: Write cleaning functions.
Step 3: Build pipeline.
Step 4: Test and validate.
Step 5: Save and reuse.
Mini summary: Build an automated cleaning project.
How to Build a Cleaning Pipeline in 6 Steps:
This module focuses on automation and efficiency. Teachers should emphasize the importance of writing reusable code. Use examples of functions and pipelines. Encourage students to build their own pipelines for different datasets. Highlight the benefits of automation for real-world data cleaning.
Parents can help children understand automation by showing them how they can automate simple tasks, like sorting emails or organizing files. Encourage children to think about how they can use code to make their daily tasks easier.
Pipeline Flowchart
Data -> Remove Duplicates -> Fill Missing -> Standardize Text -> Clean Data
Function Diagram
+-------------------+
| Function |
| Input -> Step -> |
| Output |
+-------------------+
Manual vs Automated Cleaning
| Manual | Automated |
|---|---|
| Takes hours | Takes minutes |
| Can have errors | Fewer errors |
| Not reusable | Reusable |
Function vs Pipeline
| Function | Pipeline |
|---|---|
| A single step | Multiple steps |
| One task | Sequence of tasks |
In Module Six, we learned how to automate data cleaning using functions and pipelines. We wrote functions to perform cleaning tasks and chained them together using pipe(). We added parameters to make functions flexible. We also learned how to test pipelines, handle errors, and schedule them for regular use. Automation saves time, reduces errors, and makes your workflow efficient. You are now ready to build your own automated data cleaning pipelines!
Match the term with its description:
| Term | Description |
|---|---|
| 1. Function | A. A sequence of steps |
| 2. Pipeline | B. A reusable block of code |
| 3. pipe() | C. Chains functions |
| 4. Parameter | D. A variable passed to a function |
| 5. Scheduling | E. Running code at set times |
Answers: 1-B, 2-A, 3-C, 4-D, 5-E
Scenario 1: You receive a new dataset every week. The data always has duplicates and missing values. How would you automate the cleaning process?
Scenario 2: You have a dataset with inconsistent city names and extra spaces. Write a function to clean it.
Scenario 3: You want to build a pipeline that removes duplicates, fills missing values, and standardizes text. How would you do it?
In groups, build a cleaning pipeline for a dataset of your choice. Each group member writes one cleaning function. Combine them into a pipeline and test it. Present your pipeline to the class.
Write a function that cleans a dataset by removing duplicates, filling missing values, and standardizing text. Test it on a sample dataset. Write a short report on your results.
Create an "Automated Data Cleaning Pipeline" for a dataset of your choice. Include functions for duplicates, missing values, text standardization, and outliers. Test it on a sample dataset and save the pipeline for reuse.
Find a messy dataset. Build a cleaning pipeline for it. Write a report explaining your pipeline, including the functions and the cleaning steps. Show the cleaned dataset.
Build a pipeline that automatically cleans a new dataset every week. Use scheduling to run it at a set time. Write a script that loads, cleans, and saves the data automatically.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Seven, we will learn how to validate and test data quality after cleaning. You will learn how to use asserts, data profiling, and quality checks. Get ready to ensure your data is perfect!
End of Module Six – You are now an automation expert!
Welcome to Module Seven! In all the previous modules, we learned how to clean data—removing duplicates, fixing missing values, and more. But how do we know our data is really clean? How do we make sure we didn't make any mistakes? In this module, we will learn about data quality, validation, and testing. These are the final checks to make sure your data is perfect. By the end of this module, you will be able to test your data for quality and be confident in your cleaning. Let's begin!
After this module, you will be able to:
A school in Nigeria needed to send a report to the government. The principal was nervous. What if the data had errors? What if some numbers were wrong? A data analyst said: "Don't worry. I will validate the data."
She used Python to check the data. She made sure there were no missing values. She checked that all ages were between 5 and 18. She verified that totals added up correctly. She even wrote tests to catch any mistakes. The report was perfect. The government trusted the data.
This module teaches you how to validate your data and build trust in your work.
Definition: Data quality means that data is accurate, complete, consistent, and reliable.
Why it is important: Low-quality data leads to wrong decisions. High-quality data leads to good decisions.
Simple explanation: Good data is like clean water—safe and ready to use. Bad data is like dirty water—you don't want to drink it!
Real-life example: A hospital uses high-quality patient data to give the right treatment.
School example: A teacher uses high-quality grades to know which students need help.
Home example: You use a clean budget to manage your money.
Nigerian example: A Nigerian government uses high-quality census data for planning.
Illustration:
High Quality Data = Accurate + Complete + Consistent
Mini summary: Data quality means data is accurate and reliable.
Definition: Dimensions of data quality are the different aspects that make data good: accuracy, completeness, consistency, timeliness, and validity.
Why it is important: You need to check each dimension to ensure overall quality.
Simple explanation: Think of them as the 5 checks for good data.
Real-life example: A company checks that customer addresses are complete (completeness) and correct (accuracy).
School example: A teacher checks that all grades are entered (completeness) and within a valid range (validity).
Home example: You check that all expenses are recorded (completeness) and the amounts are correct (accuracy).
Nigerian example: A business checks that all sales are recorded (completeness) and prices are correct (accuracy).
Illustration:
Dimensions:
1. Accuracy
2. Completeness
3. Consistency
4. Timeliness
5. Validity
Mini summary: Data quality has five dimensions to check.
Definition: Data validation is the process of checking data to ensure it meets certain rules or standards.
Why it is important: Validation catches errors before you use the data.
Simple explanation: It's like checking your work before you submit it.
Real-life example: A bank validates that account numbers are correct.
School example: A teacher validates that student IDs are valid.
Home example: You validate that your budget totals add up.
Nigerian example: A business validates that product codes are correct.
Illustration:
Data -> Validation Rules -> Valid Data or Error
Mini summary: Validation checks if data follows rules.
Definition: An assert statement checks if a condition is true. If it's false, it raises an error.
Why it is important: Asserts help you catch errors early in your code.
Simple explanation: You say: "I assert that all ages are > 0." If any age is less than 0, Python will stop and tell you.
Real-life example: A data analyst uses asserts to check that no negative prices exist.
School example: A student uses asserts to check that grades are between 0 and 100.
Home example: You use asserts to check that expenses are positive.
Nigerian example: A business uses asserts to check that sales are positive.
Illustration:
assert data['age'].min() > 0, "Age must be positive!"
Mini summary: Assertions check conditions and raise errors if they fail.
Definition: Common validation checks include checking data types, ranges, uniqueness, and format.
Why it is important: These are the most common ways data can be wrong.
Simple explanation: You check that numbers are numbers, dates are dates, and values are within expected limits.
Real-life example: A company checks that phone numbers have 10 digits.
School example: A teacher checks that student IDs are unique.
Home example: You check that dates are in the past.
Nigerian example: A business checks that transaction amounts are positive.
Illustration:
Checks:
- Data type check
- Range check
- Uniqueness check
- Format check
Mini summary: Common validation checks catch typical errors.
Definition: Validating data types means checking that columns have the correct data type (e.g., int, float, string).
Why it is important: Wrong data types can cause errors in analysis.
Simple explanation: You check that the "age" column contains integers, not text.
Real-life example: A company checks that the "price" column is a float.
School example: A teacher checks that "grade" is an integer.
Home example: You check that "amount" is a number.
Nigerian example: A business checks that "sales" is a float.
Illustration:
assert data['age'].dtype == 'int', "Age must be integer!"
Mini summary: Validate that data types are correct.
Definition: Validating value ranges means checking that values fall within an expected minimum and maximum.
Why it is important: Out-of-range values are often errors.
Simple explanation: You check that ages are between 0 and 120.
Real-life example: A hospital checks that patient ages are reasonable.
School example: A teacher checks that grades are between 0 and 100.
Home example: You check that expenses are positive.
Nigerian example: A business checks that salaries are within a reasonable range.
Illustration:
assert data['age'].between(0, 120).all(), "Age out of range!"
Mini summary: Validate that values are within expected ranges.
Definition: Validating uniqueness means checking that values in a column are unique (no duplicates).
Why it is important: Duplicate values can cause problems, especially for IDs.
Simple explanation: You check that student IDs are all different.
Real-life example: A company checks that customer emails are unique.
School example: A teacher checks that student IDs are unique.
Home example: You check that item names are unique.
Nigerian example: A business checks that product codes are unique.
Illustration:
assert data['id'].is_unique, "IDs must be unique!"
Mini summary: Validate that key columns have unique values.
Definition: Validating format means checking that values follow a specific pattern, like email or date formats.
Why it is important: Wrong formats can cause errors.
Simple explanation: You check that email addresses have an "@" symbol.
Real-life example: A company checks that email addresses are valid.
School example: A teacher checks that dates are in the correct format.
Home example: You check that phone numbers have 10 digits.
Nigerian example: A business checks that phone numbers are valid.
Illustration:
assert data['email'].str.contains('@').all(), "Invalid email format!"
Mini summary: Validate that values follow the right format.
Definition: A validation function is a function that checks multiple validation rules on a dataset.
Why it is important: It organizes all your validation checks in one place.
Simple explanation: You write a function that runs all your checks and reports any issues.
Real-life example: A data analyst writes a validation function for new data.
School example: A student writes a validation function for a project.
Home example: You write a validation function for your budget.
Nigerian example: A business writes a validation function for sales data.
Illustration:
def validate_data(data):
assert data['age'].dtype == 'int', "Age must be integer!"
assert data['age'].between(0, 120).all(), "Age out of range!"
assert data['id'].is_unique, "IDs must be unique!"
print("All validations passed!")
Mini summary: A validation function runs all checks.
Definition: Testing your pipeline means running it on data and checking the output.
Why it is important: You need to make sure your pipeline cleans data correctly.
Simple explanation: You run your pipeline on a sample dataset and validate the output.
Real-life example: A data analyst tests a pipeline on a sample dataset.
School example: A student tests a pipeline on a small dataset.
Home example: You test your pipeline on a few days of expenses.
Nigerian example: A business tests a pipeline on a sample sales file.
Illustration:
cleaned = clean_pipeline(data)
validate_data(cleaned)
Mini summary: Test your pipeline with validation.
Definition: Documenting data quality means writing down the validation rules and checks you used.
Why it is important: Documentation helps others trust and understand your data.
Simple explanation: You write a report that explains how you validated the data.
Real-life example: A company documents data quality for compliance.
School example: A student documents data quality for a project.
Home example: You document your budget validation rules.
Nigerian example: A business documents data quality for stakeholders.
Illustration:
Data Quality Report:
- Validation rules used
- Results of checks
- Any issues found
- Actions taken
Mini summary: Document your validation process.
Definition: Handling validation failures means deciding what to do when data fails a check.
Why it is important: You need a plan for when data is not valid.
Simple explanation: If data fails, you can reject it, fix it, or flag it for review.
Real-life example: A company rejects invalid customer records.
School example: A teacher flags invalid grades for review.
Home example: You review invalid expenses.
Nigerian example: A business flags invalid sales for review.
Illustration:
If validation fails:
1. Fix the data
2. Reject the data
3. Flag for review
Mini summary: Have a plan for validation failures.
Definition: Practice applying validation techniques to a dataset.
Why it is important: Practice makes you confident in validation.
Simple explanation: You take a dataset and run all validation checks.
Real-life example: A data analyst validates a new dataset.
School example: A student validates a dataset for a project.
Home example: You validate your expense data.
Nigerian example: A student validates local data.
Illustration:
Step 1: Load data.
Step 2: Define validation rules.
Step 3: Run validation.
Step 4: Review results.
Mini summary: Practice validation.
Definition: A project where you apply all validation techniques.
Why it is important: It gives you real-world experience.
Simple explanation: You find a dataset, clean it, and validate it.
Real-life example: A data analyst validates a company dataset.
School example: A student completes a class project.
Home example: You validate your own data.
Nigerian example: A student uses a Nigerian dataset.
Illustration:
Step 1: Load data.
Step 2: Clean data.
Step 3: Validate data.
Step 4: Document results.
Step 5: Handle issues.
Mini summary: Apply validation to a real dataset.
How to Validate Data in 6 Steps:
This module is about quality assurance. Teachers should emphasize the importance of validation in real-world data work. Use examples where poor data quality led to problems. Encourage students to write validation functions for their projects. Highlight the importance of documentation and handling failures.
Parents can help children understand validation by using everyday examples—like checking that a shopping list has no duplicates or that a budget adds up correctly. Encourage children to think about how they can "validate" their own data at home.
Validation Process Flowchart
Data -> Validate -> Valid? -> Yes -> Use Data
|
No
|
Fix Data
Data Quality Dimensions
+----------------------------------+
| Accuracy | Completeness |
| Consistency| Timeliness |
| Validity |
+----------------------------------+
Validation vs Testing
| Validation | Testing |
|---|---|
| Checks data quality | Checks code functionality |
| Ensures data is correct | Ensures code is correct |
| Runs on data | Runs on code |
High vs Low Data Quality
| High Quality | Low Quality |
|---|---|
| Accurate | Inaccurate |
| Complete | Incomplete |
| Consistent | Inconsistent |
| Reliable | Unreliable |
In Module Seven, we learned about data quality, validation, and testing. We discovered that data quality has five dimensions: accuracy, completeness, consistency, timeliness, and validity. We used assertions to validate data against rules like data types, ranges, uniqueness, and format. We created validation functions to organize our checks. We also learned to test pipelines with validation, document the process, and handle validation failures. These skills ensure that our data is clean, reliable, and trustworthy. You are now a master of data quality!
Match the term with its description:
| Term | Description |
|---|---|
| 1. Data Quality | A. Checking data against rules |
| 2. Validation | B. A statement that checks a condition |
| 3. Assert | C. Data that is accurate and reliable |
| 4. Dimension | D. An aspect of data quality |
| 5. Documentation | E. Writing down the process |
Answers: 1-C, 2-A, 3-B, 4-D, 5-E
Scenario 1: You have a dataset of students with ages, grades, and emails. Write validation rules for this dataset.
Scenario 2: You clean a dataset and want to validate it. Write a validation function to check completeness and accuracy.
Scenario 3: A validation check fails. What would you do?
In groups, choose a dataset. Write validation rules for it. Create a validation function and test it on the dataset. Discuss any issues and how to handle them.
Write a validation function for a dataset of your choice. Include checks for data type, range, uniqueness, and format. Test it on a sample dataset. Write a short report on your results.
Create a "Data Quality Report" for a dataset of your choice. Include validation rules, results of checks, any issues found, and how they were handled. Show your validation function and results.
Find a messy dataset. Clean it using your pipeline. Then, write a validation function and test it on the cleaned data. Write a report on the data quality and any issues found.
Build a complete data quality framework that includes cleaning, validation, and documentation. Apply it to a dataset and produce a data quality report.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Eight, we will learn about advanced data cleaning techniques like working with text data, regular expressions, and natural language processing. You will become a true data cleaning expert!
End of Module Seven – You are now a data quality expert!
Welcome to Module Eight! In previous modules, we learned how to clean numbers, dates, and simple text. But sometimes text is very messy—it has extra spaces, strange characters, or inconsistent patterns. For advanced text cleaning, we use a powerful tool called Regular Expressions (or "regex" for short). Regex helps us find and fix patterns in text. By the end of this module, you will be able to clean even the messiest text data. Let's begin!
After this module, you will be able to:
A company in Nigeria had customer comments. They wanted to find all phone numbers in the comments. But the phone numbers were written in many ways—some with spaces, some with dashes, some with country codes. It was impossible to find them manually.
A data analyst used regular expressions. She wrote a pattern that matched phone numbers in any format—with or without spaces, with or without country code. The regex found all the phone numbers instantly.
This module teaches you how to use regular expressions to clean messy text like a pro.
Definition: Regular expressions (regex) are patterns used to match and manipulate text.
Why it is important: Regex allows you to find, extract, or replace text based on patterns.
Simple explanation: Think of regex as a super-powered search. It can find text that follows a pattern, like all email addresses or phone numbers.
Real-life example: A company uses regex to extract all email addresses from a customer list.
School example: A student uses regex to find all dates in a document.
Home example: You use regex to find all amounts in your budget.
Nigerian example: A business uses regex to extract phone numbers from customer comments.
Illustration:
Text: "My phone is 080-123-4567"
Regex pattern: \d{3}-\d{3}-\d{4}
Found: 080-123-4567
Mini summary: Regex is a powerful tool to find patterns in text.
Definition: The re module is Python's built-in library for regular expressions.
Why it is important: You need to import re to use regex in Python.
Simple explanation: You write: import re
Real-life example: A data analyst imports re before using regex.
School example: A student imports re for a project.
Home example: You import re to clean your data.
Nigerian example: A developer imports re for text processing.
Illustration:
import re
Mini summary: Import re to use regular expressions.
Definition: Basic regex patterns include characters like \d for digits, \w for word characters, and \s for spaces.
Why it is important: These patterns help you match different types of characters.
Simple explanation: \d matches any digit (0-9). \w matches any letter, digit, or underscore. \s matches any space.
Real-life example: You use \d to find all numbers in a text.
School example: You use \w to find all words.
Home example: You use \s to find spaces.
Nigerian example: You use \d to find phone digits.
Illustration:
\d -> matches any digit
\w -> matches any word character
\s -> matches any whitespace
Mini summary: Basic patterns match digits, words, and spaces.
Definition: Quantifiers specify how many times a pattern should appear—like +, *, and ?.
Why it is important: They help you match patterns of varying length.
Simple explanation: + means "one or more", * means "zero or more", ? means "zero or one".
Real-life example: You use \d+ to match one or more digits.
School example: You use \s* to match zero or more spaces.
Home example: You use \w? to match zero or one word character.
Nigerian example: You use \d{3} to match exactly 3 digits.
Illustration:
+ -> one or more
* -> zero or more
? -> zero or one
{3} -> exactly 3
{2,4} -> 2 to 4
Mini summary: Quantifiers control how many times a pattern appears.
Definition: Special characters like ., *, +, ?, etc., have special meanings in regex. To match them literally, you need to escape them with a backslash (\).
Why it is important: If you want to match a period (.) as a period, you must escape it.
Simple explanation: To match a literal dot, you use \.
Real-life example: You use \. to match a period in a file name.
School example: You use \* to match an asterisk.
Home example: You use \? to match a question mark.
Nigerian example: You use \. to match a decimal point.
Illustration:
. -> any character (except newline)
\. -> a literal period
* -> zero or more
\* -> a literal asterisk
Mini summary: Escape special characters to match them literally.
Definition: re.search() searches for a pattern in a string and returns the first match.
Why it is important: It's useful when you just need to know if a pattern exists.
Simple explanation: re.search(pattern, text) returns a match object if found, else None.
Real-life example: You check if a text contains an email address.
School example: You check if a document contains a date.
Home example: You check if a note contains a phone number.
Nigerian example: You check if a comment contains a phone number.
Illustration:
result = re.search(r'\d{3}-\d{3}-\d{4}', text)
if result:
print("Found:", result.group())
Mini summary: re.search() finds the first match of a pattern.
Definition: re.findall() returns all non-overlapping matches of a pattern in a string.
Why it is important: It's useful when you want to extract all occurrences of a pattern.
Simple explanation: re.findall(pattern, text) returns a list of all matches.
Real-life example: You extract all email addresses from a text.
School example: You extract all dates from a document.
Home example: You extract all numbers from a budget.
Nigerian example: You extract all phone numbers from customer comments.
Illustration:
emails = re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', text)
Mini summary: re.findall() returns all matches.
Definition: re.sub() replaces occurrences of a pattern with a replacement string.
Why it is important: It's useful for cleaning text by removing or replacing unwanted patterns.
Simple explanation: re.sub(pattern, replacement, text) returns the text with replacements.
Real-life example: You remove extra spaces from text.
School example: You replace abbreviations with full words.
Home example: You remove special characters from a string.
Nigerian example: You replace inconsistent date formats.
Illustration:
clean_text = re.sub(r'\s+', ' ', text) # Replace multiple spaces with one
Mini summary: re.sub() replaces patterns in text.
Definition: re.split() splits a string by the occurrences of a pattern.
Why it is important: It's useful for breaking text into parts based on a delimiter.
Simple explanation: re.split(pattern, text) returns a list of split parts.
Real-life example: You split a sentence by commas.
School example: You split a CSV line by commas.
Home example: You split a grocery list by commas.
Nigerian example: You split customer data by semicolons.
Illustration:
parts = re.split(r',\s*', text)
Mini summary: re.split() splits text by a pattern.
Definition: You can use regex in Pandas with methods like str.contains(), str.extract(), and str.replace().
Why it is important: It allows you to clean text data in DataFrames efficiently.
Simple explanation: You use data['col'].str.contains(pattern) to find rows matching a pattern.
Real-life example: You extract all phone numbers from a customer column.
School example: You extract all student IDs.
Home example: You extract all amounts from a budget column.
Nigerian example: You extract all email addresses from a customer list.
Illustration:
data['phone'] = data['comment'].str.extract(r'(\d{3}-\d{3}-\d{4})')
Mini summary: Use regex in Pandas with str methods.
Definition: Common tasks include removing extra spaces, extracting phone numbers, cleaning emails, and removing special characters.
Why it is important: These are the most frequent cleaning tasks in real-world data.
Simple explanation: You use regex to find and fix these common problems.
Real-life example: A company removes extra spaces from customer names.
School example: A student extracts dates from a text.
Home example: You remove special characters from a list.
Nigerian example: A business extracts phone numbers from comments.
Illustration:
# Remove extra spaces
data['text'] = data['text'].str.replace(r'\s+', ' ', regex=True)
# Extract phone numbers
data['phone'] = data['text'].str.extract(r'(\d{3}-\d{3}-\d{4})')
Mini summary: Regex handles common text cleaning tasks.
Definition: Complex patterns combine multiple elements to match intricate text patterns.
Why it is important: Some patterns are not simple—like matching emails or URLs.
Simple explanation: You combine characters, quantifiers, and groups to build a pattern.
Real-life example: A pattern to match Nigerian phone numbers.
School example: A pattern to match student IDs.
Home example: A pattern to match dates.
Nigerian example: A pattern to match Lagos addresses.
Illustration:
# Nigerian phone number pattern
pattern = r'(0|\+234)[7-9][0-9]{9}'
Mini summary: Complex patterns combine many elements.
Definition: Practice applying regex to clean messy text.
Why it is important: Practice helps you master regex.
Simple explanation: You take a messy text column and apply regex to clean it.
Real-life example: A data analyst cleans customer comments.
School example: A student cleans a text dataset.
Home example: You clean your own text data.
Nigerian example: A student cleans local text data.
Illustration:
data['clean_text'] = data['text'].str.replace(r'[^a-zA-Z0-9\s]', '', regex=True)
Mini summary: Practice regex on real text data.
Definition: You can use regex to validate text patterns, like checking if a string is a valid email or phone number.
Why it is important: It ensures text data meets expected formats.
Simple explanation: You check if a string matches a pattern.
Real-life example: A company validates email addresses.
School example: A student validates student IDs.
Home example: You validate phone numbers.
Nigerian example: A business validates NIN numbers.
Illustration:
def is_valid_email(email):
pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
return re.match(pattern, email) is not None
Mini summary: Regex validates text patterns.
Definition: A project where you apply regex to clean a text dataset.
Why it is important: It gives you real-world experience.
Simple explanation: You find a messy text dataset and clean it using regex.
Real-life example: A data analyst cleans customer comments.
School example: A student completes a class project.
Home example: You clean your own text data.
Nigerian example: A student uses a Nigerian text dataset.
Illustration:
Step 1: Load text data.
Step 2: Identify patterns.
Step 3: Apply regex to clean.
Step 4: Validate results.
Mini summary: Apply regex to a real text cleaning project.
How to Clean Text with Regex in 6 Steps:
This module introduces regular expressions, a powerful but sometimes tricky topic. Teachers should start with simple patterns and gradually introduce complexity. Use plenty of examples and practice exercises. Emphasize that regex is a tool for pattern matching, and it's okay to start with simple patterns.
Parents can help children understand regex by using everyday examples—like searching for a specific word in a book or finding numbers in a text. Encourage children to think about patterns in text and how they can be used to clean data.
Regex Matching Process
Text -> Pattern -> Match? -> Yes -> Extract/Replace
|
No
|
No Action
Common Regex Patterns
\d -> digit
\w -> word character
\s -> whitespace
. -> any character
+ -> one or more
* -> zero or more
? -> zero or one
{3} -> exactly 3
Regex Methods
| Method | Purpose |
|---|---|
| re.search() | Find first match |
| re.findall() | Find all matches |
| re.sub() | Replace matches |
| re.split() | Split text |
Without Regex vs With Regex
| Without Regex | With Regex |
|---|---|
| Manual string methods | Powerful pattern matching |
| Limited to simple tasks | Handles complex patterns |
| Slower for large data | Efficient for large data |
In Module Eight, we learned about regular expressions (regex) for advanced text cleaning. We discovered what regex is and how to use it to find patterns in text. We learned basic patterns, quantifiers, and how to escape special characters. We used re.search(), re.findall(), re.sub(), and re.split() to manipulate text. We also applied regex in Pandas using str methods. We practiced common text cleaning tasks like removing extra spaces, extracting phone numbers, and validating email addresses. These skills are essential for cleaning messy text data. You are now a text-cleaning expert!
Match the term with its description:
| Term | Description |
|---|---|
| 1. re.search() | A. Finds all matches |
| 2. re.findall() | B. Replaces matches |
| 3. re.sub() | C. Finds first match |
| 4. re.split() | D. Splits text |
| 5. \d | E. Matches a digit |
Answers: 1-C, 2-A, 3-B, 4-D, 5-E
Scenario 1: You have a dataset of customer comments. Extract all phone numbers using regex.
Scenario 2: You have a dataset of addresses. Clean the addresses by removing extra spaces and special characters.
Scenario 3: You have a dataset of emails. Validate that all emails are in the correct format.
In groups, find a messy text dataset. Identify common patterns (phone numbers, emails, dates). Write regex patterns to extract and clean these patterns. Present your results to the class.
Write a Python script that uses regex to clean a text column in a dataset. Include extraction, replacement, and validation. Test it on a sample dataset. Write a short report on your results.
Create a "Text Cleaning Tool" that uses regex to clean a text column. Include functions to remove extra spaces, extract phone numbers, and validate emails. Apply it to a real dataset and show the results.
Find a messy text dataset (e.g., customer comments, social media posts). Clean it using regex. Write a report on your cleaning process, including the patterns you used and the results.
Write a regex pattern to match Nigerian phone numbers in all common formats (with spaces, dashes, country code). Test it on a large text dataset and extract all phone numbers.
Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.
In Module Nine, we will learn about data integration and merging from multiple sources. You will learn how to combine data from different files and databases. Get ready for the final module!
End of Module Eight – You are now a regex master!