🐍 Python for Data Cleaning

Course Outline – beginner friendly, hands-on, and practical

⏳ ~6 weeks 🧠 no prior coding required 📊 real datasets

📌 Course overview

Data cleaning is one of the most important steps in any data project. This course will teach you how to use Python to find, fix, and prevent common data problems – missing values, duplicates, inconsistent formats, outliers, and more. By the end, you will be able to prepare raw data for analysis, dashboards, or machine learning.

🎯 Goal: write clean Python code to clean, reshape, and validate real-world data (CSV, Excel, JSON).

📚 Course modules

🛠️ Tools & libraries covered

• pandas – data manipulation and cleaning
• numpy – numerical operations
• matplotlib / seaborn – quick visual checks (optional)
• Jupyter Notebook – interactive coding
• CSV, Excel, and JSON import/export

🧠 Who this course is for

• Complete beginners who know basic Python (or are learning it)
• Anyone who works with spreadsheets and wants to upgrade to Python
• Students, analysts, and professionals who want to save time on data prep

📈 What you will be able to do after this course

• ✅ Load and explore any CSV or Excel file with confidence
• ✅ Identify and fix missing, incorrect, or messy data
• ✅ Remove duplicates, standardize text, and handle outliers
• ✅ Merge, aggregate, and reshape data for analysis
• ✅ Create reusable Python scripts to clean new datasets quickly

📘 Module Two: Python for Data Cleaning – Pandas and Data Exploration

Welcome to Module Two! In the first module, we learned about data cleaning and the basics of Python. Now, we will learn about the most important tool for data cleaning in Python—Pandas. Pandas is like a superpower for working with data. It helps us load, explore, and clean data easily. In this module, we will focus on loading data and exploring it. By the end, you will be able to look at any dataset and understand its structure, types, and problems. Let's get started!

🎯 Learning Objectives

After this module, you will be able to:

• Install and import Pandas in Python.
• Load data from CSV files into Pandas.
• Use Pandas to view and summarize data.
• Understand data types and basic statistics.
• Identify missing values and duplicates.

📖 Warm-up Story: The Supermarket Data

A supermarket in Lagos wanted to understand its customers better. They had a huge file with all sales records. But the file was too big to read on paper. They needed a tool to explore the data quickly.

Their data analyst used Pandas. With just a few lines of code, she loaded the data, looked at the first few rows, checked for missing values, and summarized the sales. She found that many customers bought rice and beans together. She recommended a combo offer, and sales increased.

This module teaches you the skills that analyst used—loading data, exploring it, and understanding it. These are the first steps to cleaning and analyzing any dataset.

📚 Main Lessons

Lesson 1: What is Pandas?

Definition: Pandas is a Python library that gives us tools to work with data.

Why it is important: Pandas makes data cleaning and analysis easy and fast.

Simple explanation: Think of Pandas as a magical toolbox for data. You put your data in, and you can do all kinds of things with it.

Real-life example: Data analysts use Pandas to clean and analyze data for companies.

School example: Students use Pandas to analyze survey data.

Home example: You can use Pandas to analyze your family's expenses.

Nigerian example: Nigerian businesses use Pandas to understand customer behavior.

Illustration:

    Python + Pandas = Powerful Data Cleaning
    

Mini summary: Pandas is a Python library for working with data.

Lesson 2: Installing Pandas

Definition: Installing Pandas means adding it to your Python environment.

Why it is important: You need Pandas installed to use it.

Simple explanation: It's like downloading a new app on your phone.

Real-life example: A data scientist installs Pandas before starting a project.

School example: A student installs Pandas in their computer lab.

Home example: You install Pandas on your personal computer.

Nigerian example: A Nigerian developer installs Pandas to build a data app.

Illustration:

    pip install pandas
    

Mini summary: Install Pandas using pip install pandas.

Lesson 3: Importing Pandas

Definition: Importing means telling Python that you want to use Pandas.

Why it is important: You must import Pandas before using it.

Simple explanation: It's like opening a toolbox before using a tool.

Real-life example: You write: import pandas as pd

School example: A student writes the import command in their notebook.

Home example: You write the import command in your code.

Nigerian example: A developer imports Pandas to start cleaning data.

Illustration:

    import pandas as pd
    

Mini summary: Import Pandas with "import pandas as pd".

Lesson 4: Loading Data with Pandas

Definition: Loading data means reading a data file into Pandas.

Why it is important: You need to load data to work with it.

Simple explanation: You use pd.read_csv() to load a CSV file.

Real-life example: A data analyst loads sales data from a CSV file.

School example: A student loads a dataset for a class project.

Home example: You load your family's expense data from Excel.

Nigerian example: A Nigerian business loads customer data for analysis.

Illustration:

    data = pd.read_csv("sales.csv")
    

Mini summary: Use pd.read_csv() to load CSV files.

Lesson 5: Viewing Data with head()

Definition: head() is a function that shows the first few rows of data.

Why it is important: It gives you a quick look at the data.

Simple explanation: Like looking at the first page of a book.

Real-life example: An analyst uses head() to check the data format.

School example: A student uses head() to see the dataset.

Home example: You use head() to look at your expense data.

Nigerian example: A Nigerian business uses head() to inspect customer records.

Illustration:

    data.head()
    

Mini summary: head() shows the first 5 rows of data.

Lesson 6: Viewing Data with tail()

Definition: tail() shows the last few rows of data.

Why it is important: It helps you see the end of the data.

Simple explanation: Like looking at the last page of a book.

Real-life example: An analyst checks the last few records.

School example: A student uses tail() to see the last rows.

Home example: You check the last entries in your expense log.

Nigerian example: A business checks the last transactions.

Illustration:

    data.tail()
    

Mini summary: tail() shows the last 5 rows of data.

Lesson 7: Understanding Data Types

Definition: Data types are the types of data in each column, like numbers, text, or dates.

Why it is important: Knowing data types helps you clean data correctly.

Simple explanation: Numbers are int or float, text is object, dates are datetime.

Real-life example: An analyst checks data types before analysis.

School example: A student uses info() to check data types.

Home example: You check if your expense amounts are numbers.

Nigerian example: A business checks data types in customer records.

Illustration:

    data.dtypes
    

Mini summary: Use dtypes to check data types.

Lesson 8: Getting Information with info()

Definition: info() gives a summary of the data, including column names, types, and non-null counts.

Why it is important: It gives a complete overview of the data.

Simple explanation: Like a summary sheet for your data.

Real-life example: An analyst uses info() to understand the dataset.

School example: A student uses info() to describe the dataset.

Home example: You use info() to check your family budget.

Nigerian example: A business uses info() to understand customer data.

Illustration:

    data.info()
    

Mini summary: info() gives a summary of the data.

Lesson 9: Descriptive Statistics with describe()

Definition: describe() gives statistics like mean, min, max, and count for numerical columns.

Why it is important: It helps you understand the distribution of numbers.

Simple explanation: Like a report card for your numbers.

Real-life example: An analyst uses describe() to see average sales.

School example: A student uses describe() to analyze test scores.

Home example: You use describe() to see average spending.

Nigerian example: A business uses describe() to understand transaction amounts.

Illustration:

    data.describe()
    

Mini summary: describe() gives statistics for numerical data.

Lesson 10: Checking for Missing Values

Definition: Missing values are empty or null values in the data.

Why it is important: Missing values can cause errors in analysis.

Simple explanation: You use isnull() and sum() to count missing values.

Real-life example: An analyst checks for missing customer emails.

School example: A student checks for missing assignment scores.

Home example: You check for missing expense entries.

Nigerian example: A business checks for missing customer data.

Illustration:

    data.isnull().sum()
    

Mini summary: Use isnull().sum() to count missing values.

Lesson 11: Checking for Duplicates

Definition: Duplicates are rows that appear more than once.

Why it is important: Duplicates can distort analysis.

Simple explanation: You use duplicated() and sum() to count duplicates.

Real-life example: An analyst checks for duplicate customer records.

School example: A student checks for duplicate survey responses.

Home example: You check for duplicate purchases.

Nigerian example: A business checks for duplicate transactions.

Illustration:

    data.duplicated().sum()
    

Mini summary: Use duplicated().sum() to count duplicates.

Lesson 12: Exploring Data with Value Counts

Definition: value_counts() counts how many times each unique value appears in a column.

Why it is important: It helps you understand categorical data.

Simple explanation: Like counting how many students are in each grade.

Real-life example: An analyst counts the number of customers in each city.

School example: A student counts the number of students in each class.

Home example: You count how many times you bought each item.

Nigerian example: A business counts the number of customers in each state.

Illustration:

    data['city'].value_counts()
    

Mini summary: value_counts() counts unique values in a column.

Lesson 13: Grouping Data with groupby()

Definition: groupby() groups data by a column and allows you to perform calculations.

Why it is important: It helps you summarize data by categories.

Simple explanation: Like grouping students by class and finding the average score.

Real-life example: An analyst groups sales by city.

School example: A student groups test scores by subject.

Home example: You group expenses by category.

Nigerian example: A business groups sales by region.

Illustration:

    data.groupby('city').mean()
    

Mini summary: groupby() groups data for summary analysis.

Lesson 14: Sorting Data with sort_values()

Definition: sort_values() sorts data by a column in ascending or descending order.

Why it is important: It helps you find the highest or lowest values.

Simple explanation: Like sorting test scores from highest to lowest.

Real-life example: An analyst sorts sales by amount.

School example: A student sorts grades to find the best and worst.

Home example: You sort expenses to see the largest amounts.

Nigerian example: A business sorts customers by purchase amount.

Illustration:

    data.sort_values('sales', ascending=False)
    

Mini summary: sort_values() sorts data by a column.

Lesson 15: Your First Data Exploration Project

Definition: A data exploration project is a complete process of loading, viewing, and summarizing a dataset.

Why it is important: It gives you hands-on practice with Pandas.

Simple explanation: You load a dataset, view it, check types, get statistics, and find missing values.

Real-life example: A data analyst explores a new dataset before cleaning.

School example: A student explores a class dataset.

Home example: You explore your family's expense data.

Nigerian example: A Nigerian business explores customer data.

Illustration:

    data = pd.read_csv("file.csv")
    data.head()
    data.info()
    data.describe()
    data.isnull().sum()
    data.duplicated().sum()
    

Mini summary: Practice exploring data with Pandas.

🔑 Key Vocabulary

• Pandas: A Python library for data work.
• CSV: A file format for data.
• DataFrame: The main Pandas data structure.
• head(): Shows the first rows.
• tail(): Shows the last rows.
• info(): Shows data summary.
• describe(): Shows statistics.
• isnull(): Checks for missing values.
• duplicated(): Checks for duplicates.
• groupby(): Groups data.
• sort_values(): Sorts data.
• value_counts(): Counts unique values.

💡 Important Concepts

• Pandas is a powerful tool for data.
• Load data with read_csv().
• Explore data with head(), tail(), info(), describe().
• Check for missing values with isnull().sum().
• Check for duplicates with duplicated().sum().
• Analyze categories with value_counts().
• Group data with groupby() and sort with sort_values().

📝 Step-by-Step Explanations

How to Explore Data with Pandas in 6 Steps:

1. Install Pandas: pip install pandas
2. Import Pandas: import pandas as pd
3. Load Data: data = pd.read_csv("file.csv")
4. View Data: data.head(), data.tail()
5. Summarize Data: data.info(), data.describe()
6. Check Problems: data.isnull().sum(), data.duplicated().sum()

🌍 Real-life Examples

• A data analyst loads a sales dataset and checks for missing values.
• A student uses Pandas to explore a survey dataset.
• A business uses groupby() to understand sales by region.
• A researcher uses describe() to understand patient data.

🇳🇬 Nigerian Examples

• A Nigerian business uses Pandas to explore customer data from Lagos.
• A Nigerian school uses Pandas to analyze student attendance.
• A Nigerian farmer uses Pandas to explore crop yield data.
• A Nigerian bank uses Pandas to explore transaction data.

🎈 Fun Examples Children Relate To

• Use Pandas to explore a list of video game scores.
• Use Pandas to explore your friends' favorite foods.
• Use Pandas to explore a list of movies you like.
• Use Pandas to explore your school's sports scores.

🏠 Everyday Examples

• Use Pandas to explore your family's monthly expenses.
• Use Pandas to explore a list of your books.
• Use Pandas to explore your daily to-do list.
• Use Pandas to explore your family's grocery list.

🧑‍🏫 Teacher Notes

This module introduces Pandas, the most important library for data cleaning in Python. Teachers should emphasize the importance of exploration before cleaning. Encourage students to practice with different datasets. Use hands-on exercises where students load, view, and summarize data. Highlight the practical use of Pandas in Nigeria and around the world.

👪 Parent Tips

Parents can help children by finding simple datasets to explore. For example, a list of monthly expenses or a list of books. Encourage children to practice loading and exploring data. Show them how data exploration is used in real life, like checking a budget or analyzing a shopping list.

🌟 Interesting Facts

• Pandas was created by Wes McKinney in 2008.
• The name "Pandas" comes from "Panel Data".
• Pandas is used by companies like Google, Facebook, and Twitter.
• Pandas is one of the most popular Python libraries.

🤔 Did You Know?

• Did you know that Pandas can handle data millions of rows?
• Did you know that Pandas has built-in functions for cleaning data?
• Did you know that Pandas can read data from many formats—CSV, Excel, JSON, SQL?
• Did you know that many Nigerian companies use Pandas for data analysis?

🧠 Remember This

• Pandas is a Python library for data work.
• Load data with pd.read_csv().
• View data with head() and tail().
• Summarize data with info() and describe().
• Check for missing values with isnull().sum().
• Check for duplicates with duplicated().sum().
• Analyze categories with value_counts().
• Group data with groupby() and sort with sort_values().

❌ Common Mistakes

• Forgetting to import Pandas: Always import pandas as pd.
• Wrong file path: Make sure the file path is correct.
• Not using head(): Always look at the data first.
• Ignoring missing values: Always check for missing values.
• Not checking data types: Use dtypes to check types.
• Forgetting to sort: Use sort_values() to order data.

✅ Best Practices

• Always import Pandas with "import pandas as pd".
• Use head() and tail() to view data.
• Use info() and describe() to summarize data.
• Check for missing values and duplicates.
• Use value_counts() to understand categories.
• Use groupby() for grouped analysis.
• Use sort_values() to order data.
• Practice with different datasets.

📊 Clear ASCII Illustrations

Pandas Workflow

    Load Data -> View Data -> Summarize Data -> Check Problems -> Clean Data
    

DataFrame Structure

    +---------+---------+---------+
    | Name    | Age     | City    |
    +---------+---------+---------+
    | Chidi   | 12      | Lagos   |
    | Ama     | 14      | Abuja   |
    | Bola    | 13      | Ibadan  |
    +---------+---------+---------+
    

📋 Comparison Tables

head() vs tail()

info() vs describe()

📌 End-of-Module Summary

In Module Two, we learned about Pandas, the most important Python library for data cleaning. We installed and imported Pandas. We learned how to load data from CSV files. We explored data using head(), tail(), info(), and describe(). We checked for missing values and duplicates. We also learned about value_counts(), groupby(), and sort_values(). These are the essential skills for exploring and understanding any dataset. You are now ready to start cleaning data with Pandas!

❓ Frequently Asked Questions

1. What is Pandas? A Python library for data work.
2. How do I install Pandas? Use "pip install pandas".
3. How do I load a CSV file? Use "pd.read_csv("file.csv")".
4. What does head() do? Shows the first few rows.
5. What does info() do? Gives a summary of the data.
6. What does describe() do? Gives statistics for numerical columns.
7. How do I check for missing values? Use "data.isnull().sum()".
8. How do I check for duplicates? Use "data.duplicated().sum()".
9. What does groupby() do? Groups data for summary analysis.
10. What does sort_values() do? Sorts data by a column.

📝 Review Questions

1. What is Pandas?
2. How do you install Pandas?
3. What command do you use to import Pandas?
4. How do you load a CSV file?
5. What does head() show?
6. What does tail() show?
7. What does info() show?
8. What does describe() show?
9. How do you check for missing values?
10. How do you check for duplicates?
11. What does value_counts() do?
12. What does groupby() do?
13. What does sort_values() do?
14. Give a Nigerian example of using Pandas.
15. What is the first step in exploring data?

✍️ Fill-in-the-Blank

1. ______ is a Python library for data work. (Pandas)
2. Use ______ to install Pandas. (pip install pandas)
3. Use ______ to import Pandas. (import pandas as pd)
4. Use ______ to load a CSV file. (pd.read_csv())
5. ______ shows the first few rows. (head())
6. ______ gives a summary of the data. (info())
7. ______ gives statistics for numerical columns. (describe())
8. Use ______ to check for missing values. (isnull().sum())
9. Use ______ to check for duplicates. (duplicated().sum())
10. ______ groups data for summary analysis. (groupby())

✅ True or False

1. Pandas is a Python library for data work. (True)
2. You don't need to import Pandas to use it. (False)
3. head() shows the last few rows. (False)
4. info() gives a summary of the data. (True)
5. describe() gives statistics for text columns. (False)
6. isnull().sum() counts missing values. (True)
7. duplicated().sum() counts duplicates. (True)
8. value_counts() counts unique values. (True)
9. groupby() groups data. (True)
10. sort_values() sorts data. (True)

🔘 Multiple Choice Questions

1. What is Pandas?
   A) A type of animal
   B) A Python library for data
   C) A programming language
   D) A file format
   Answer: B
2. How do you install Pandas?
   A) import pandas
   B) pip install pandas
   C) install pandas
   D) pandas install
   Answer: B
3. What command imports Pandas?
   A) import pandas as pd
   B) import pandas
   C) as pd import pandas
   D) pandas import
   Answer: A
4. How do you load a CSV file?
   A) data.read_csv()
   B) pd.read_csv()
   C) read.csv()
   D) csv.read()
   Answer: B
5. What does head() show?
   A) First rows
   B) Last rows
   C) Summary
   D) Statistics
   Answer: A
6. What does tail() show?
   A) First rows
   B) Last rows
   C) Summary
   D) Statistics
   Answer: B
7. What does info() show?
   A) First rows
   B) Last rows
   C) Summary
   D) Statistics
   Answer: C
8. What does describe() show?
   A) First rows
   B) Last rows
   C) Summary
   D) Statistics
   Answer: D
9. How do you check for missing values?
   A) data.missing()
   B) data.isnull().sum()
   C) data.null()
   D) data.empty()
   Answer: B
10. How do you check for duplicates?
    A) data.duplicate()
    B) data.duplicated().sum()
    C) data.double()
    D) data.repeat()
    Answer: B
11. What does value_counts() do?
    A) Counts unique values
    B) Counts rows
    C) Counts missing values
    D) Counts duplicates
    Answer: A
12. What does groupby() do?
    A) Groups data
    B) Sorts data
    C) Counts data
    D) Summarizes data
    Answer: A
13. What does sort_values() do?
    A) Groups data
    B) Sorts data
    C) Counts data
    D) Summarizes data
    Answer: B
14. Which is a Nigerian example of using Pandas?
    A) Analyzing Lagos customer data
    B) Analyzing New York sales
    C) Analyzing London traffic
    D) Analyzing Paris weather
    Answer: A
15. What is the first step in exploring data?
    A) Clean data
    B) Load data
    C) Analyze data
    D) Delete data
    Answer: B

🔗 Matching Exercises

Match the function with its description:

Answers: 1-C, 2-A, 3-B, 4-E, 5-D

✏️ Short Answer Questions

1. What is Pandas?
2. How do you load a CSV file?
3. What does info() show?
4. How do you check for missing values?
5. Give a Nigerian example of using Pandas.

🎭 Scenario-based Exercises

Scenario 1: You have a CSV file of student grades. Load it and use head() to view the data. Then use info() and describe() to summarize it.

Scenario 2: You have a sales dataset. Check for missing values and duplicates. Count the number of missing values in each column.

Scenario 3: You have a customer dataset. Use value_counts() to see how many customers are from each city.

👥 Group Activity

In groups, download a public dataset (e.g., from Kaggle or data.gov.ng). Load it into Pandas. Each group member takes turns exploring the data using head(), info(), describe(), and checking for missing values. Present your findings to the class.

🧑 Individual Activity

Create a simple dataset (at least 10 rows and 3 columns) on a topic of your choice (e.g., friends, books, movies). Save it as a CSV file. Write Python code to load, explore, and summarize the data.

💬 Classroom Discussion Questions

• Why is it important to explore data before cleaning?
• What are some challenges you faced when loading or exploring data?
• How can Pandas help in your daily life?
• What are some real-world datasets you would like to explore?
• How can Nigerian businesses benefit from Pandas?

🛠️ Mini Project

Create a "Data Exploration Report" for a dataset of your choice. Include: data loading, first and last rows, summary, data types, missing values, duplicates, and at least one groupby analysis. Present your report in a Jupyter Notebook.

📄 Practical Assignment

Download a dataset from a Nigerian data portal (e.g., data.gov.ng). Load it into Pandas. Write a Python script that explores the data and prints: the first 5 rows, the last 5 rows, a summary, the number of missing values, and the number of duplicates.

🏆 Challenge Exercise

Find a dataset with at least 5 columns. Use Pandas to explore it. Then, write a short report (in plain English) describing what the data is about, what you observed, and any potential problems you found (like missing values or inconsistencies).

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Pandas is a powerful library for data work.
• Load data with pd.read_csv().
• Explore data with head(), tail(), info(), describe().
• Check for missing values and duplicates.
• Use value_counts(), groupby(), and sort_values() for deeper analysis.
• Always explore data before cleaning.

🔜 Preparation for Module Three

In Module Three, we will learn how to handle missing values—one of the most common data cleaning tasks. We will practice filling, dropping, and imputing missing values. Get ready to clean your first dataset!

End of Module Two – You are now a Pandas explorer!

📘 Module Three: Python for Data Cleaning – Handling Missing Values

Welcome to Module Three! In the first two modules, we learned about data cleaning and how to explore data with Pandas. Now, we will tackle one of the most common problems in data cleaning—missing values. Missing values are like empty spaces in your data. They can cause problems if not handled properly. In this module, you will learn how to find, count, and fix missing values using Python. By the end, you will be able to clean datasets that have missing information. Let's get started!

🎯 Learning Objectives

After this module, you will be able to:

• Identify and count missing values in a dataset.
• Decide whether to drop or fill missing values.
• Use dropna() to remove rows with missing values.
• Use fillna() to fill missing values with a specific value.
• Use interpolation to fill missing values logically.
• Apply these techniques to real datasets.

📖 Warm-up Story: The Incomplete Survey

A school in Nigeria conducted a survey about students' favorite subjects. But some students didn't answer all the questions. Some left the age field blank. Others didn't say their favorite subject. The school principal wanted to know the top subject, but the missing answers made it hard.

A data analyst used Python to fix the problem. First, she found all the missing values. Then, she filled in the missing ages with the average age of the class. She dropped the rows where the favorite subject was missing because there were only a few. Now the data was clean, and she could see that Mathematics was the most popular subject.

This module will teach you how to handle missing values just like that analyst.

📚 Main Lessons

Lesson 1: What are Missing Values?

Definition: Missing values are empty or null values in a dataset. They are like blank spaces on a form.

Why it is important: Missing values can make your analysis wrong. You need to fix them before using the data.

Simple explanation: Imagine a class register where some students' names are missing. You can't know how many students are in the class.

Real-life example: A customer's phone number is missing in a database.

School example: A student forgot to write their age on a form.

Home example: You forgot to write down one expense in your budget.

Nigerian example: A farmer forgot to record rainfall on one day.

Illustration:

    +---------+---------+
    | Name    | Age     |
    +---------+---------+
    | Chidi   | 12      |
    | Ama     |         |
    | Bola    | 13      |
    +---------+---------+
    

Mini summary: Missing values are empty spaces in data.

Lesson 2: Why Missing Values Happen

Definition: Missing values happen for many reasons—people forget, data isn't collected, or there are errors.

Why it is important: Understanding why data is missing helps you decide how to fix it.

Simple explanation: Sometimes people skip questions. Sometimes data is lost.

Real-life example: A customer didn't provide their email address.

School example: A student didn't answer a question on a survey.

Home example: You forgot to record a receipt.

Nigerian example: A bank may have missing data due to system errors.

Illustration:

    Reasons for Missing Values:
    1. People skip questions.
    2. Data entry errors.
    3. System failures.
    4. Not applicable.
    

Mini summary: Missing values happen for various reasons.

Lesson 3: Finding Missing Values in Pandas

Definition: You use Pandas functions like isnull() and notnull() to find missing values.

Why it is important: You need to know where the missing values are.

Simple explanation: isnull() shows True for missing values and False for non-missing.

Real-life example: A data analyst uses isnull() to find empty fields.

School example: A student uses isnull() to check for missing grades.

Home example: You use isnull() to find missing expenses.

Nigerian example: A business uses isnull() to find missing customer data.

Illustration:

    data.isnull()
    

Mini summary: isnull() helps find missing values.

Lesson 4: Counting Missing Values

Definition: You can count missing values using isnull().sum().

Why it is important: Counting helps you understand how much data is missing.

Simple explanation: isnull().sum() counts the missing values in each column.

Real-life example: An analyst counts missing customer emails.

School example: A student counts missing test scores.

Home example: You count missing expenses.

Nigerian example: A business counts missing transaction data.

Illustration:

    data.isnull().sum()
    

Mini summary: isnull().sum() counts missing values.

Lesson 5: The Problem with Missing Values

Definition: Missing values can cause errors in calculations and lead to wrong conclusions.

Why it is important: If you ignore missing values, your analysis might be wrong.

Simple explanation: If you calculate the average age and some ages are missing, the average will be wrong.

Real-life example: A company might target the wrong customers.

School example: A teacher might think students are doing well when they are not.

Home example: You might think you spent less money than you actually did.

Nigerian example: A government might plan for the wrong population.

Illustration:

    Dirty Data -> Wrong Analysis -> Bad Decisions
    

Mini summary: Missing values lead to wrong conclusions.

Lesson 6: Dealing with Missing Values – Drop or Fill?

Definition: You can either drop (remove) missing values or fill (replace) them with a value.

Why it is important: The choice depends on the data and the problem.

Simple explanation: If only a few rows are missing, you can drop them. If many rows are missing, you might want to fill them.

Real-life example: If a customer is missing an email, you might fill it with a default.

School example: If a student is missing a grade, you might fill it with the class average.

Home example: If you missed a receipt, you might estimate the amount.

Nigerian example: If a farmer missed a day of rainfall, you might fill it with an average.

Illustration:

    Missing Values -> Drop or Fill?
    

Mini summary: Decide whether to drop or fill missing values.

Lesson 7: Dropping Missing Values with dropna()

Definition: dropna() removes rows or columns that have missing values.

Why it is important: It's a quick way to clean data when there are few missing values.

Simple explanation: If a row has any missing value, it is deleted.

Real-life example: A data analyst drops rows with missing customer emails.

School example: A student drops rows with missing grades.

Home example: You drop missing expenses from your budget.

Nigerian example: A business drops rows with missing transaction details.

Illustration:

    data.dropna()
    

Mini summary: dropna() removes rows with missing values.

Lesson 8: Filling Missing Values with fillna()

Definition: fillna() replaces missing values with a specified value.

Why it is important: It allows you to keep all the data and still fix missing values.

Simple explanation: You replace missing values with a number, a text, or an average.

Real-life example: You fill missing customer emails with "unknown@email.com".

School example: You fill missing grades with the class average.

Home example: You fill missing expenses with 0.

Nigerian example: You fill missing crop data with last year's average.

Illustration:

    data.fillna(0)
    

Mini summary: fillna() replaces missing values with a value.

Lesson 9: Filling with the Mean, Median, or Mode

Definition: You can fill missing values with the mean (average), median (middle value), or mode (most common value).

Why it is important: These are logical ways to fill missing numbers.

Simple explanation: If you know the average age, you can fill missing ages with that average.

Real-life example: A company fills missing salaries with the median salary.

School example: You fill missing test scores with the class average.

Home example: You fill missing grocery costs with the average.

Nigerian example: A farmer fills missing rainfall data with the monthly average.

Illustration:

    data.fillna(data.mean())
    

Mini summary: Use mean, median, or mode to fill missing values.

Lesson 10: Forward Fill and Backward Fill

Definition: Forward fill (ffill) uses the previous value to fill a missing value. Backward fill (bfill) uses the next value.

Why it is important: Useful for time series data where the next or previous value makes sense.

Simple explanation: If Monday's data is missing, you use Sunday's data.

Real-life example: Stock prices are often filled using forward fill.

School example: If attendance is missing for one day, you use the previous day.

Home example: If you missed a daily expense, you use the previous day's expense.

Nigerian example: A farmer uses forward fill for missing daily temperature data.

Illustration:

    data.fillna(method='ffill')
    

Mini summary: Forward and backward fill use neighboring values.

Lesson 11: Interpolation

Definition: Interpolation estimates missing values based on the values around them.

Why it is important: It creates a smooth estimate for missing data.

Simple explanation: If you have points on a line, interpolation fills the missing point on the line.

Real-life example: Scientists use interpolation to fill missing climate data.

School example: You can interpolate missing scores based on previous and next scores.

Home example: You can interpolate missing temperatures.

Nigerian example: A farmer uses interpolation to estimate missing crop growth data.

Illustration:

    data.interpolate()
    

Mini summary: Interpolation estimates missing values smoothly.

Lesson 12: When to Drop vs. When to Fill

Definition: You drop when missing data is very little. You fill when missing data is significant.

Why it is important: The wrong choice can affect your analysis.

Simple explanation: If 1% of data is missing, drop it. If 20% is missing, fill it.

Real-life example: A company drops less than 5% missing data and fills more than that.

School example: A teacher drops a few missing grades but fills many.

Home example: You drop missing receipts that are few, but fill missing expenses that are many.

Nigerian example: A business drops a few missing records but fills many to keep the dataset.

Illustration:

    Little Missing -> Drop
    Lots Missing -> Fill
    

Mini summary: Drop when little is missing; fill when much is missing.

Lesson 13: Handling Missing Values in Categorical Data

Definition: Categorical data is text data, like "city" or "gender".

Why it is important: You can't use mean for text. You can use mode or fill with "Unknown".

Simple explanation: If the city is missing, fill it with "Unknown" or the most common city.

Real-life example: A company fills missing customer city with "Unknown".

School example: A student fills missing subject with "Unknown".

Home example: You fill missing item category with "Other".

Nigerian example: A business fills missing region with "Unknown".

Illustration:

    data['city'].fillna('Unknown')
    

Mini summary: Fill categorical missing values with "Unknown" or mode.

Lesson 14: Practice Cleaning Missing Values

Definition: Practice applying all the techniques you learned.

Why it is important: Practice helps you remember and build skills.

Simple explanation: You take a messy dataset and fix all missing values.

Real-life example: A data analyst practices on a sample dataset.

School example: A student works on a class exercise.

Home example: You practice on your own expense data.

Nigerian example: A Nigerian student uses local data for practice.

Illustration:

    data.dropna()
    data.fillna(value)
    data.fillna(data.mean())
    data.fillna(method='ffill')
    data.interpolate()
    

Mini summary: Practice all techniques to master missing values.

Lesson 15: Your Missing Data Project

Definition: A project where you apply all missing value techniques to a real dataset.

Why it is important: It gives you real-world experience.

Simple explanation: You find a dataset, explore it, find missing values, and fix them.

Real-life example: A data analyst cleans a customer dataset.

School example: A student completes a class project.

Home example: You clean your family's expense data.

Nigerian example: A student uses a Nigerian dataset for the project.

Illustration:

    Step 1: Load data.
    Step 2: Explore data.
    Step 3: Find missing values.
    Step 4: Decide to drop or fill.
    Step 5: Apply the chosen method.
    Step 6: Validate the cleaned data.
    

Mini summary: Apply missing value techniques to a real dataset.

🔑 Key Vocabulary

• Missing Values: Empty spaces in data.
• isnull(): Finds missing values.
• sum(): Counts missing values.
• dropna(): Removes missing values.
• fillna(): Fills missing values.
• Mean: Average value.
• Median: Middle value.
• Mode: Most common value.
• Forward Fill: Uses previous value.
• Backward Fill: Uses next value.
• Interpolation: Estimates missing values.
• Categorical: Text or category data.

💡 Important Concepts

• Missing values are common.
• Find missing values with isnull().sum().
• Drop missing values with dropna().
• Fill missing values with fillna().
• Use mean, median, or mode to fill numbers.
• Use forward/backward fill for time data.
• Use interpolation for smooth estimates.
• For text, fill with "Unknown" or mode.
• Drop when little missing; fill when lots missing.

📝 Step-by-Step Explanations

How to Handle Missing Values in 6 Steps:

1. Load Data: data = pd.read_csv("file.csv")
2. Explore Data: data.head(), data.info()
3. Find Missing: data.isnull().sum()
4. Decide: Will you drop or fill?
5. Apply: data.dropna() or data.fillna(value)
6. Validate: data.isnull().sum() again

🌍 Real-life Examples

• A hospital fills missing patient records with "Unknown".
• A bank drops rows with missing transaction amounts.
• A school fills missing grades with the class average.
• A company fills missing customer emails with "none@example.com".

🇳🇬 Nigerian Examples

• A Nigerian bank fills missing customer addresses with "Unknown".
• A Nigerian school fills missing attendance with the previous day.
• A Nigerian farmer uses interpolation to fill missing crop data.
• A Nigerian business drops rows with missing product details.

🎈 Fun Examples Children Relate To

• Missing values are like missing pieces in a puzzle.
• Filling missing values is like drawing in the missing parts of a picture.
• Dropping missing values is like removing a damaged puzzle piece.
• Using the average is like guessing a missing number.

🏠 Everyday Examples

• If you missed a day in your journal, you might leave it blank or fill it with "no entry".
• If you don't know the exact amount of a receipt, you might estimate it.
• If you forget to write a task on your to-do list, you might add it later.
• If you don't know the exact date of an event, you might use "unknown".

🧑‍🏫 Teacher Notes

This module is critical for data cleaning. Teachers should emphasize that handling missing values is a decision-making process. Encourage students to think about why data is missing and what method is best. Use real datasets with missing values. Have students practice dropping, filling with mean, forward fill, and interpolation. Discuss the pros and cons of each method.

👪 Parent Tips

Parents can help children understand missing values by using everyday examples. For instance, "If we don't know the price of one item, we can guess based on similar items." Encourage children to think about how they deal with missing information in daily life.

🌟 Interesting Facts

• Missing values are also called null values, NA (not available), or NaN (not a number).
• In some datasets, up to 50% of the data can be missing.
• Many machine learning algorithms cannot handle missing values, so you must clean them first.
• Interpolation is often used in weather forecasting.

🤔 Did You Know?

• Did you know that missing values can be intentionally left blank to represent "not applicable"?
• Did you know that some data analysts use "NULL" to represent missing values?
• Did you know that Pandas uses NaN to represent missing values?
• Did you know that missing data can be predicted using machine learning?

🧠 Remember This

• Missing values are empty spaces in data.
• Find them with isnull().sum().
• Drop them with dropna().
• Fill them with fillna().
• Use mean, median, mode for numbers.
• Use forward/backward fill for time data.
• Use interpolation for smooth estimates.
• For text, fill with "Unknown" or mode.
• Drop when little missing; fill when lots missing.

❌ Common Mistakes

• Not checking for missing values: Always check first.
• Dropping too much data: Don't drop if many rows are missing.
• Filling with the wrong value: Use appropriate values.
• Using mean for text data: You can't use mean on text.
• Not validating: Always check that missing values are fixed.
• Ignoring the reason for missing data: Understanding why helps choose the right method.

✅ Best Practices

• Always check for missing values first.
• Understand why data is missing.
• Drop only when few rows are missing.
• Fill with appropriate values (mean, median, mode).
• Use forward/backward fill for time series.
• Use interpolation for smooth data.
• For text, use "Unknown" or mode.
• Always validate after cleaning.
• Document your approach.

📊 Clear ASCII Illustrations

Missing Value Handling Flowchart

    Missing Values -> Check -> Drop or Fill?
    -> Drop: dropna() -> Clean Data
    -> Fill: fillna() -> Clean Data
    

Forward Fill Example

    Original: 10, 20, NaN, 40
    Forward Fill: 10, 20, 20, 40
    

📋 Comparison Tables

Drop vs Fill

Fill Methods

📌 End-of-Module Summary

In Module Three, we learned how to handle missing values in Python using Pandas. We learned to find and count missing values with isnull().sum(). We explored two main approaches: dropping missing values with dropna() and filling them with fillna(). We filled using mean, median, mode, forward/backward fill, and interpolation. We also discussed how to handle categorical data and when to drop vs. fill. These skills are essential for cleaning real-world datasets. You are now ready to fix missing values in any dataset!

❓ Frequently Asked Questions

1. What are missing values? Empty spaces in data.
2. How do you find missing values? Use data.isnull().sum().
3. What does dropna() do? Removes rows with missing values.
4. What does fillna() do? Fills missing values with a value.
5. What is mean? The average value.
6. What is median? The middle value.
7. What is mode? The most common value.
8. What is forward fill? Uses the previous value.
9. What is interpolation? Estimates missing values.
10. When should you drop vs. fill? Drop when little missing; fill when lots missing.

📝 Review Questions

1. What are missing values?
2. How do you find missing values in Pandas?
3. What does dropna() do?
4. What does fillna() do?
5. What is the mean?
6. What is the median?
7. What is the mode?
8. What is forward fill?
9. What is interpolation?
10. When should you drop missing values?
11. When should you fill missing values?
12. How do you fill missing values in categorical data?
13. What is the first step in handling missing values?
14. Why is it important to validate after cleaning?
15. Give a Nigerian example of handling missing values.

✍️ Fill-in-the-Blank

1. ______ are empty spaces in data. (Missing values)
2. Use ______ to find missing values. (isnull().sum())
3. ______ removes rows with missing values. (dropna())
4. ______ fills missing values with a value. (fillna())
5. The ______ is the average value. (mean)
6. The ______ is the middle value. (median)
7. The ______ is the most common value. (mode)
8. ______ uses the previous value to fill a missing value. (Forward fill)
9. ______ estimates missing values smoothly. (Interpolation)
10. For categorical data, fill with ______. (Unknown or mode)

✅ True or False

1. Missing values are not a problem. (False)
2. isnull().sum() counts missing values. (True)
3. dropna() removes all missing values. (True)
4. fillna() cannot be used for text data. (False)
5. The mean is the middle value. (False)
6. The mode is the most common value. (True)
7. Forward fill uses the next value. (False)
8. Interpolation is used for smooth data. (True)
9. You should always drop missing values. (False)
10. You should always validate after cleaning. (True)

🔘 Multiple Choice Questions

1. What are missing values?
   A) Extra data
   B) Empty spaces in data
   C) Duplicate data
   D) Correct data
   Answer: B
2. How do you find missing values?
   A) data.count()
   B) data.isnull().sum()
   C) data.missing()
   D) data.empty()
   Answer: B
3. What does dropna() do?
   A) Removes missing values
   B) Fills missing values
   C) Counts missing values
   D) Finds missing values
   Answer: A
4. What does fillna() do?
   A) Removes missing values
   B) Fills missing values
   C) Counts missing values
   D) Finds missing values
   Answer: B
5. What is the mean?
   A) Middle value
   B) Most common value
   C) Average value
   D) Largest value
   Answer: C
6. What is the median?
   A) Average value
   B) Middle value
   C) Most common value
   D) Largest value
   Answer: B
7. What is the mode?
   A) Average value
   B) Middle value
   C) Most common value
   D) Largest value
   Answer: C
8. What is forward fill?
   A) Uses the next value
   B) Uses the previous value
   C) Uses the average
   D) Uses the median
   Answer: B
9. What is interpolation?
   A) Removes missing values
   B) Estimates missing values
   C) Counts missing values
   D) Fills with a constant
   Answer: B
10. When should you drop missing values?
    A) When many are missing
    B) When few are missing
    C) When data is categorical
    D) When data is numerical
    Answer: B
11. When should you fill missing values?
    A) When few are missing
    B) When many are missing
    C) When data is categorical
    D) When data is numerical
    Answer: B
12. How do you fill missing values in categorical data?
    A) With mean
    B) With median
    C) With mode or "Unknown"
    D) With interpolation
    Answer: C
13. What is the first step in handling missing values?
    A) Fill them
    B) Drop them
    C) Find and count them
    D) Ignore them
    Answer: C
14. Why validate after cleaning?
    A) To make sure it's done
    B) To check if missing values are fixed
    C) To delete data
    D) To add more missing values
    Answer: B
15. Which is a Nigerian example of handling missing values?
    A) Filling missing rainfall data
    B) Dropping missing student records
    C) Both A and B
    D) Neither A nor B
    Answer: C

🔗 Matching Exercises

Match the term with its description:

Answers: 1-C, 2-A, 3-B, 4-E, 5-D

✏️ Short Answer Questions

1. What are missing values?
2. How do you count missing values?
3. What is the difference between dropna() and fillna()?
4. When would you use forward fill?
5. Give a Nigerian example of using interpolation.

🎭 Scenario-based Exercises

Scenario 1: You have a dataset of student grades. Some grades are missing. What would you do? Explain your steps.

Scenario 2: A company's customer data has missing emails for 60% of customers. What would you do?

Scenario 3: A farmer has daily rainfall data with some missing days. How would you fill the missing days?

👥 Group Activity

In groups, get a dataset with missing values (from Kaggle or data.gov.ng). Each group member uses a different method to handle missing values (drop, fill with mean, forward fill, interpolation). Compare the results and discuss which method worked best.

🧑 Individual Activity

Create a small dataset with missing values. Write Python code to handle the missing values using at least three different methods. Compare the results and write a short reflection on your choice.

💬 Classroom Discussion Questions

• Why is it important to handle missing values?
• What are some challenges in handling missing values?
• When would you choose to drop missing values?
• When would you choose to fill missing values?
• How can Nigerian businesses benefit from handling missing values?

🛠️ Mini Project

Create a "Missing Values Report" for a dataset of your choice. Include: the number of missing values per column, your chosen method to handle them, and the cleaned data. Show before and after using head() and info().

📄 Practical Assignment

Download a dataset with missing values (e.g., from data.gov.ng). Handle all missing values using appropriate methods. Write a report explaining your choices and show the cleaned dataset.

🏆 Challenge Exercise

Find a dataset with missing values in both numerical and categorical columns. Use different methods for each type. Write a Python script that automates the handling of missing values.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Missing values are empty spaces in data.
• Find missing values with isnull().sum().
• Drop missing values with dropna().
• Fill missing values with fillna().
• Use mean, median, mode for numbers.
• Use forward/backward fill for time data.
• Use interpolation for smooth estimates.
• For text, fill with "Unknown" or mode.
• Drop when little missing; fill when lots missing.

🔜 Preparation for Module Four

In Module Four, we will learn how to handle duplicate data, inconsistent formatting, and outliers. These are the next steps in cleaning messy data. Get ready to become a data cleaning expert!

End of Module Three – You are now a missing values master!

📘 Module Four: Python for Data Cleaning – Duplicates, Inconsistencies, and Outliers

Welcome to Module Four! In the previous modules, we learned how to handle missing values. Now, we will tackle three more common data problems: duplicates, inconsistencies, and outliers. Duplicates are repeated rows. Inconsistencies are data that doesn't match—like "Lagos" and "lagos". Outliers are numbers that are very different from the rest—like an age of 150. By the end of this module, you will be able to find and fix all these problems. Let's begin!

🎯 Learning Objectives

After this module, you will be able to:

• Identify and remove duplicate rows.
• Standardize text data to fix inconsistencies.
• Find and handle outliers using the IQR method.
• Apply these techniques to real datasets.

📖 Warm-up Story: The Messy Customer List

A business in Nigeria had a list of customers. But the list was messy. Some customers appeared twice. Some names were spelled differently—"Chidi" and "Chidi". Some ages were wrong—like 200 years old!

The business owner wanted to send a newsletter, but the duplicates meant some people would get it twice. The wrong ages made it hard to understand the customer base. A data analyst cleaned the list. She removed duplicates, fixed spellings, and removed the crazy ages. The list became clean and useful.

This module teaches you how to do the same.

📚 Main Lessons

Lesson 1: What are Duplicates?

Definition: Duplicates are rows that appear more than once in a dataset.

Why it is important: Duplicates can make your analysis wrong. They can count the same customer twice.

Simple explanation: Imagine a class register where a student's name appears twice. That would be a duplicate.

Real-life example: A customer is listed twice in a database.

School example: A student's name appears twice on a list.

Home example: You write the same item twice on a shopping list.

Nigerian example: A company has duplicate customer records.

Illustration:

    +---------+---------+
    | Name    | Age     |
    +---------+---------+
    | Chidi   | 12      |
    | Ama     | 14      |
    | Chidi   | 12      |  <- Duplicate
    +---------+---------+
    

Mini summary: Duplicates are repeated rows.

Lesson 2: Finding Duplicates

Definition: You use duplicated() to find duplicate rows.

Why it is important: You need to know where the duplicates are.

Simple explanation: duplicated() returns True for duplicate rows.

Real-life example: A data analyst uses duplicated() to find repeated customer records.

School example: A student uses duplicated() to find repeated names.

Home example: You use duplicated() to find repeated expenses.

Nigerian example: A business uses duplicated() to find duplicate orders.

Illustration:

    data.duplicated()
    

Mini summary: duplicated() finds duplicate rows.

Lesson 3: Counting Duplicates

Definition: You can count duplicates using duplicated().sum().

Why it is important: It tells you how many duplicates you have.

Simple explanation: duplicated().sum() counts all duplicate rows.

Real-life example: An analyst counts duplicate customer emails.

School example: A student counts duplicate names.

Home example: You count duplicate items.

Nigerian example: A business counts duplicate sales records.

Illustration:

    data.duplicated().sum()
    

Mini summary: duplicated().sum() counts duplicates.

Lesson 4: Removing Duplicates

Definition: You use drop_duplicates() to remove duplicate rows.

Why it is important: It cleans the data by removing repeated rows.

Simple explanation: drop_duplicates() removes all duplicate rows and keeps only the first occurrence.

Real-life example: A company removes duplicate customers.

School example: A student removes duplicate names.

Home example: You remove duplicate items from your shopping list.

Nigerian example: A business removes duplicate orders.

Illustration:

    data.drop_duplicates()
    

Mini summary: drop_duplicates() removes duplicates.

Lesson 5: What are Inconsistencies?

Definition: Inconsistencies are data that doesn't match. For example, "Lagos" and "lagos" are inconsistent.

Why it is important: Inconsistencies can cause errors in analysis.

Simple explanation: If some cities are written in uppercase and others in lowercase, they won't be counted as the same city.

Real-life example: A city is spelled "Abuja" and "abuja".

School example: A subject is written as "Math" and "math".

Home example: "Milk" and "milk" on a shopping list.

Nigerian example: "Lagos" and "lagos" in a customer database.

Illustration:

    +---------+
    | City    |
    +---------+
    | Lagos   |
    | lagos   |
    | LAGOS   |
    +---------+
    

Mini summary: Inconsistencies are data that doesn't match.

Lesson 6: Standardizing Text Data

Definition: Standardizing means making text data uniform. For example, converting all text to lowercase.

Why it is important: It ensures that "Lagos" and "lagos" are treated the same.

Simple explanation: You use str.lower() to convert text to lowercase.

Real-life example: A company standardizes city names.

School example: A student standardizes subject names.

Home example: You standardize your shopping list.

Nigerian example: A business standardizes customer city names.

Illustration:

    data['city'] = data['city'].str.lower()
    

Mini summary: Standardize text using str.lower().

Lesson 7: Removing Whitespace

Definition: Whitespace is extra spaces before or after text.

Why it is important: Extra spaces can cause inconsistencies.

Simple explanation: You use str.strip() to remove leading and trailing spaces.

Real-life example: A company removes extra spaces in customer names.

School example: A student removes extra spaces in a list.

Home example: You remove extra spaces in your notes.

Nigerian example: A business removes spaces in product names.

Illustration:

    data['name'] = data['name'].str.strip()
    

Mini summary: Use str.strip() to remove extra spaces.

Lesson 8: Replacing Values

Definition: You can use replace() to change values in a column.

Why it is important: It helps fix wrong spellings or inconsistent values.

Simple explanation: You replace "lagos" with "Lagos".

Real-life example: A company replaces "N/A" with "Unknown".

School example: A student replaces "Math" with "Mathematics".

Home example: You replace "milk" with "Milk".

Nigerian example: A business replaces "Abuja" with "Abuja" (capitalize).

Illustration:

    data['city'] = data['city'].replace('lagos', 'Lagos')
    

Mini summary: Use replace() to fix values.

Lesson 9: Using str Methods for Cleaning

Definition: Pandas has many string methods to clean text data.

Why it is important: They make it easy to fix text data.

Simple explanation: Methods like str.lower(), str.strip(), str.replace().

Real-life example: A data analyst uses str methods to clean text.

School example: A student uses str methods to clean survey data.

Home example: You use str methods to clean a grocery list.

Nigerian example: A business uses str methods to clean product names.

Illustration:

    data['name'] = data['name'].str.title()
    

Mini summary: Use str methods to clean text.

Lesson 10: What are Outliers?

Definition: Outliers are values that are very different from the rest of the data.

Why it is important: Outliers can skew your analysis and lead to wrong conclusions.

Simple explanation: If most students are 12-14 years old, a student who is 150 years old is an outlier.

Real-life example: A salary of 1 billion naira is an outlier.

School example: A test score of 200 when the maximum is 100.

Home example: A grocery bill of 500,000 naira.

Nigerian example: A transaction amount of 10 million naira in a small business.

Illustration:

    +---------+
    | Age     |
    +---------+
    | 12      |
    | 13      |
    | 14      |
    | 150     |  <- Outlier
    +---------+
    

Mini summary: Outliers are extreme values.

Lesson 11: Finding Outliers with IQR

Definition: IQR (Interquartile Range) is a method to find outliers. It looks at the middle 50% of data.

Why it is important: It's a standard way to identify outliers.

Simple explanation: You calculate Q1 (25th percentile) and Q3 (75th percentile). Then IQR = Q3 - Q1. Any value below Q1 - 1.5*IQR or above Q3 + 1.5*IQR is an outlier.

Real-life example: A data analyst uses IQR to find salary outliers.

School example: A student uses IQR to find grade outliers.

Home example: You use IQR to find spending outliers.

Nigerian example: A business uses IQR to find transaction outliers.

Illustration:

    Q1 = data['age'].quantile(0.25)
    Q3 = data['age'].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5*IQR
    upper = Q3 + 1.5*IQR
    outliers = data[(data['age'] < lower) | (data['age'] > upper)]
    

Mini summary: IQR is a method to find outliers.

Lesson 12: Handling Outliers – Remove or Cap?

Definition: You can remove outliers or cap them to a maximum or minimum value.

Why it is important: Removing outliers can lose important data. Capping keeps the data but limits extreme values.

Simple explanation: You can drop outliers or replace them with the upper limit.

Real-life example: A company caps salaries at a certain limit.

School example: A teacher caps grades at 100.

Home example: You cap your spending at a limit.

Nigerian example: A bank caps transaction amounts.

Illustration:

    Remove: data = data[(data['age'] >= lower) & (data['age'] <= upper)]
    Cap: data['age'] = data['age'].clip(lower, upper)
    

Mini summary: Remove or cap outliers.

Lesson 13: Visualizing Outliers with Boxplots

Definition: A boxplot is a chart that shows the distribution of data and highlights outliers.

Why it is important: It helps you see outliers visually.

Simple explanation: The boxplot shows the median, quartiles, and outliers as dots.

Real-life example: A data analyst uses a boxplot to find salary outliers.

School example: A student uses a boxplot to see grade distribution.

Home example: You use a boxplot to see spending distribution.

Nigerian example: A business uses a boxplot to see sales distribution.

Illustration:

    import matplotlib.pyplot as plt
    plt.boxplot(data['age'])
    plt.show()
    

Mini summary: Boxplots help visualize outliers.

Lesson 14: Practice Cleaning Duplicates, Inconsistencies, and Outliers

Definition: Practice applying all the techniques you learned.

Why it is important: Practice helps you remember and build skills.

Simple explanation: You take a messy dataset and clean all three types of problems.

Real-life example: A data analyst cleans a customer dataset.

School example: A student works on a class exercise.

Home example: You clean your own data.

Nigerian example: A Nigerian student uses local data.

Illustration:

    Remove duplicates: data.drop_duplicates()
    Standardize text: data['city'] = data['city'].str.lower().str.strip()
    Remove outliers: using IQR method
    

Mini summary: Practice all three techniques.

Lesson 15: Your Clean Data Project

Definition: A project where you apply all techniques to a real dataset.

Why it is important: It gives you real-world experience.

Simple explanation: You find a messy dataset and clean it completely.

Real-life example: A data analyst cleans a company dataset.

School example: A student completes a class project.

Home example: You clean your own expense data.

Nigerian example: A student uses a Nigerian dataset.

Illustration:

    Step 1: Load data.
    Step 2: Explore data.
    Step 3: Remove duplicates.
    Step 4: Fix inconsistencies.
    Step 5: Handle outliers.
    Step 6: Validate the cleaned data.
    

Mini summary: Apply all techniques to a real dataset.

🔑 Key Vocabulary

• Duplicate: A repeated row.
• Inconsistency: Data that doesn't match.
• Outlier: An extreme value.
• duplicated(): Finds duplicates.
• drop_duplicates(): Removes duplicates.
• str.lower(): Converts text to lowercase.
• str.strip(): Removes extra spaces.
• replace(): Changes values.
• IQR: Interquartile Range, used to find outliers.
• Boxplot: A chart to visualize outliers.

💡 Important Concepts

• Duplicates are repeated rows.
• Remove duplicates with drop_duplicates().
• Inconsistencies are data that doesn't match.
• Standardize text with str.lower() and str.strip().
• Outliers are extreme values.
• Use IQR to find outliers.
• Remove or cap outliers.
• Boxplots help visualize outliers.

📝 Step-by-Step Explanations

How to Clean Duplicates, Inconsistencies, and Outliers in 6 Steps:

1. Load Data: data = pd.read_csv("file.csv")
2. Find Duplicates: data.duplicated().sum()
3. Remove Duplicates: data.drop_duplicates(inplace=True)
4. Fix Inconsistencies: data['col'] = data['col'].str.lower().str.strip()
5. Find Outliers: Use IQR method.
6. Handle Outliers: Remove or cap them.

🌍 Real-life Examples

• A company removes duplicate customer records.
• A school standardizes city names in student data.
• A bank removes outlier transaction amounts.
• A hospital standardizes patient names.

🇳🇬 Nigerian Examples

• A Nigerian business removes duplicate sales records.
• A Nigerian school standardizes student names.
• A Nigerian bank removes outlier transaction amounts.
• A Nigerian government standardizes local government names.

🎈 Fun Examples Children Relate To

• Removing duplicates from a list of your favorite games.
• Standardizing a list of names so they all start with a capital letter.
• Finding outliers in a list of ages—like a 100-year-old!
• Cleaning a list of your favorite foods.

🏠 Everyday Examples

• Removing duplicate items from a shopping list.
• Standardizing a list of cities to all lowercase.
• Finding outliers in your spending—like a huge expense.
• Cleaning your to-do list by removing repeated tasks.

🧑‍🏫 Teacher Notes

This module addresses three key data cleaning tasks. Teachers should emphasize the importance of each task with practical examples. Use datasets that contain duplicates, inconsistencies, and outliers. Show students how to use Pandas methods and the IQR technique. Encourage students to think about whether to remove or cap outliers.

👪 Parent Tips

Parents can help children understand these concepts by using everyday examples. For instance, "When we organize the pantry, we remove duplicate items and make sure all labels are consistent." Encourage children to think about how they clean and organize data in their daily lives.

🌟 Interesting Facts

• Duplicates are often caused by human error.
• Inconsistencies are very common in large datasets.
• Outliers can sometimes be the most interesting data points.
• The IQR method for finding outliers was developed by John Tukey.

🤔 Did You Know?

• Did you know that duplicates can be hidden and hard to find?
• Did you know that inconsistent data is a major problem for companies?
• Did you know that outliers can sometimes be errors, but other times they are important discoveries?
• Did you know that boxplots are a standard way to visualize outliers?

🧠 Remember This

• Duplicates are repeated rows.
• Remove duplicates with drop_duplicates().
• Inconsistencies are data that doesn't match.
• Standardize text with str.lower() and str.strip().
• Outliers are extreme values.
• Use IQR to find outliers.
• Remove or cap outliers.
• Boxplots help visualize outliers.

❌ Common Mistakes

• Not removing duplicates: This can lead to wrong counts.
• Ignoring inconsistencies: "Lagos" and "lagos" are treated as different.
• Not handling outliers: They can skew your analysis.
• Capping outliers too aggressively: Can remove important information.
• Using wrong IQR multiplier: 1.5 is standard, but sometimes 3 is used.
• Not validating after cleaning.

✅ Best Practices

• Always check for duplicates first.
• Standardize text data early.
• Use IQR to identify outliers.
• Decide whether to remove or cap outliers based on context.
• Visualize outliers with boxplots.
• Always validate after cleaning.
• Document your approach.

📊 Clear ASCII Illustrations

Duplicate Removal Flowchart

    Data -> Find Duplicates -> Remove Duplicates -> Clean Data
    

Boxplot Illustration

    +---------------------+
    |   *     Outlier     |
    |                     |
    |   +-------+         |
    |   |       |         |
    |   | Box   |         |
    |   |       |         |
    |   +-------+         |
    |   *     Outlier     |
    +---------------------+
    

📋 Comparison Tables

Remove vs Cap Outliers

str.lower() vs str.strip()

📌 End-of-Module Summary

In Module Four, we tackled three more data cleaning tasks: duplicates, inconsistencies, and outliers. We learned how to find and remove duplicates with duplicated() and drop_duplicates(). We fixed inconsistencies by standardizing text with str.lower(), str.strip(), and replace(). We identified outliers using the IQR method and decided whether to remove or cap them. We also visualized outliers using boxplots. These are essential skills for cleaning any real-world dataset. You are now ready to clean data like a professional!

❓ Frequently Asked Questions

1. What are duplicates? Repeated rows in data.
2. How do you remove duplicates? Use drop_duplicates().
3. What are inconsistencies? Data that doesn't match.
4. How do you fix text inconsistencies? Use str.lower() and str.strip().
5. What are outliers? Extreme values.
6. How do you find outliers? Use the IQR method.
7. What is IQR? Interquartile Range.
8. Should you remove or cap outliers? It depends on the data and context.
9. What is a boxplot? A chart to visualize outliers.
10. Why validate after cleaning? To make sure it's done correctly.

📝 Review Questions

1. What are duplicates?
2. How do you find duplicates?
3. How do you remove duplicates?
4. What are inconsistencies?
5. How do you fix text inconsistencies?
6. What is str.strip() used for?
7. What are outliers?
8. What is the IQR method?
9. How do you find outliers using IQR?
10. What is the difference between removing and capping outliers?
11. What is a boxplot?
12. Why is it important to handle outliers?
13. Give a Nigerian example of duplicates.
14. Give a Nigerian example of an inconsistency.
15. Give a Nigerian example of an outlier.

✍️ Fill-in-the-Blank

1. ______ are repeated rows in data. (Duplicates)
2. Use ______ to remove duplicates. (drop_duplicates())
3. ______ are data that doesn't match. (Inconsistencies)
4. Use ______ to convert text to lowercase. (str.lower())
5. Use ______ to remove extra spaces. (str.strip())
6. ______ are extreme values. (Outliers)
7. ______ is used to find outliers. (IQR)
8. IQR stands for ______. (Interquartile Range)
9. You can ______ outliers or ______ them. (remove, cap)
10. A ______ helps visualize outliers. (boxplot)

✅ True or False

1. Duplicates are not a problem. (False)
2. drop_duplicates() removes duplicates. (True)
3. Inconsistencies are not important. (False)
4. str.lower() converts text to uppercase. (False)
5. str.strip() removes extra spaces. (True)
6. Outliers are normal values. (False)
7. IQR is a method to find outliers. (True)
8. You should always remove outliers. (False)
9. A boxplot shows outliers. (True)
10. You should always validate after cleaning. (True)

🔘 Multiple Choice Questions

1. What are duplicates?
   A) Unique values
   B) Repeated rows
   C) Missing values
   D) Inconsistent data
   Answer: B
2. How do you remove duplicates?
   A) dropna()
   B) drop_duplicates()
   C) fillna()
   D) duplicated()
   Answer: B
3. What are inconsistencies?
   A) Data that matches
   B) Data that doesn't match
   C) Missing values
   D) Duplicates
   Answer: B
4. How do you fix text inconsistencies?
   A) str.upper()
   B) str.lower()
   C) str.strip()
   D) All of the above
   Answer: D
5. What are outliers?
   A) Normal values
   B) Extreme values
   C) Missing values
   D) Duplicates
   Answer: B
6. What is IQR used for?
   A) Finding duplicates
   B) Finding outliers
   C) Fixing inconsistencies
   D) Removing missing values
   Answer: B
7. What does IQR stand for?
   A) Interquartile Range
   B) Internal Quality Review
   C) Integrated Query Range
   D) International Quality Report
   Answer: A
8. What is a boxplot used for?
   A) Finding duplicates
   B) Fixing inconsistencies
   C) Visualizing outliers
   D) Removing missing values
   Answer: C
9. When should you remove outliers?
   A) When many outliers
   B) When few outliers
   C) When data is categorical
   D) When data is numerical
   Answer: B
10. When should you cap outliers?
    A) When few outliers
    B) When many outliers
    C) When data is categorical
    D) When data is numerical
    Answer: B
11. What is the IQR multiplier for finding outliers?
    A) 1.0
    B) 1.5
    C) 2.0
    D) 3.0
    Answer: B
12. Which method removes leading and trailing spaces?
    A) str.lower()
    B) str.upper()
    C) str.strip()
    D) str.replace()
    Answer: C
13. Which method converts text to lowercase?
    A) str.lower()
    B) str.upper()
    C) str.strip()
    D) str.replace()
    Answer: A
14. Which method replaces values?
    A) str.lower()
    B) str.upper()
    C) str.strip()
    D) replace()
    Answer: D
15. Which is a Nigerian example of an inconsistency?
    A) "Lagos" and "lagos"
    B) Age 12 and 13
    C) Price 100 and 200
    D) Missing values
    Answer: A

🔗 Matching Exercises

Match the term with its description:

Answers: 1-A, 2-C, 3-B, 4-D, 5-E

✏️ Short Answer Questions

1. What are duplicates and how do you remove them?
2. What are inconsistencies and how do you fix them?
3. What are outliers and how do you find them?
4. When would you remove vs. cap outliers?
5. Give a Nigerian example of an outlier.

🎭 Scenario-based Exercises

Scenario 1: You have a list of customers with duplicate records. What would you do?

Scenario 2: You have a city column with "Lagos", "lagos", and "LAGOS". How would you fix it?

Scenario 3: You have a salary column with values like 50,000, 60,000, and 10,000,000. How would you handle the outlier?

👥 Group Activity

In groups, get a dataset with duplicates, inconsistencies, and outliers. Each group member takes one problem to fix. Combine your work to produce a clean dataset. Discuss the methods you used.

🧑 Individual Activity

Create a dataset with at least 10 rows, including duplicates, inconsistencies, and outliers. Write Python code to clean all three types of problems. Show your cleaned dataset.

💬 Classroom Discussion Questions

• Why are duplicates a problem?
• How can inconsistencies affect analysis?
• When is an outlier actually important?
• What is the best way to handle outliers?
• How can Nigerian businesses benefit from cleaning data?

🛠️ Mini Project

Create a "Data Cleaning Report" for a dataset of your choice. Include sections on duplicates, inconsistencies, and outliers. Show your methods and the cleaned data.

📄 Practical Assignment

Download a messy dataset. Clean it by removing duplicates, fixing inconsistencies, and handling outliers. Write a report explaining your steps and show the cleaned dataset.

🏆 Challenge Exercise

Find a dataset with all three problems. Write a Python script that automatically cleans the data. Include methods for duplicates, inconsistencies, and outliers. Test it on the dataset and show the results.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Duplicates are repeated rows.
• Remove duplicates with drop_duplicates().
• Inconsistencies are data that doesn't match.
• Standardize text with str.lower() and str.strip().
• Outliers are extreme values.
• Use IQR to find outliers.
• Remove or cap outliers.
• Boxplots help visualize outliers.
• Always validate after cleaning.

🔜 Preparation for Module Five

In Module Five, we will learn about advanced data cleaning techniques like working with dates, merging datasets, and reshaping data. You will become a true data cleaning expert!

End of Module Four – You are now a data cleaning all-star!

📘 Module Five: Python for Data Cleaning – Working with Dates, Merging, and Reshaping

Welcome to Module Five! In the previous modules, we cleaned missing values, duplicates, inconsistencies, and outliers. Now, we will learn three more important skills: working with dates, merging datasets, and reshaping data. Dates are often messy and need cleaning. Merging combines datasets. Reshaping changes how data is organized. By the end of this module, you will be able to handle almost any data cleaning task. Let's get started!

🎯 Learning Objectives

After this module, you will be able to:

• Convert and clean date data using Pandas.
• Merge and join datasets using common keys.
• Reshape data using pivot and melt.
• Apply these techniques to real datasets.

📖 Warm-up Story: The School Database

A school in Nigeria had two separate databases. One had student names and IDs. The other had their test scores. They wanted to combine them to see each student's scores. They also had a column with dates in different formats—"12-05-2024" and "2024/05/12".

A data analyst used Pandas to merge the two datasets on student ID. She also converted all dates to a single format. Finally, she reshaped the data so that each subject had its own column. Now the school had a clean, unified dataset. This module teaches you these powerful skills.

📚 Main Lessons

Lesson 1: Why Dates are Tricky

Definition: Dates can be written in many formats—like "12-05-2024", "2024/05/12", or "May 12, 2024".

Why it is important: Inconsistent dates can cause errors in analysis.

Simple explanation: If some dates are in day-month-year and others are in month-day-year, they will be wrong.

Real-life example: A company has dates in different formats.

School example: A teacher records dates in different ways.

Home example: You write dates differently on different pages.

Nigerian example: A business has dates in "DD/MM/YYYY" and "MM/DD/YYYY".

Illustration:

    12-05-2024 (DD-MM-YYYY)
    2024/05/12 (YYYY/MM/DD)
    May 12, 2024
    

Mini summary: Dates can be messy and need cleaning.

Lesson 2: Converting Dates with pd.to_datetime()

Definition: pd.to_datetime() converts a column to datetime format.

Why it is important: It standardizes dates and allows you to do date operations.

Simple explanation: You give Pandas a column of dates, and it converts them to a single, standard format.

Real-life example: A data analyst converts date strings to datetime.

School example: A teacher converts date formats.

Home example: You convert dates in your budget.

Nigerian example: A business converts dates to a standard format.

Illustration:

    data['date'] = pd.to_datetime(data['date'])
    

Mini summary: pd.to_datetime() standardizes dates.

Lesson 3: Extracting Date Components

Definition: You can extract parts of a date like year, month, day, or day of the week.

Why it is important: It helps you analyze patterns over time.

Simple explanation: You can get the month from a date and see which month has the most sales.

Real-life example: A company analyzes sales by month.

School example: A teacher tracks attendance by day of the week.

Home example: You track expenses by month.

Nigerian example: A business analyzes sales by quarter.

Illustration:

    data['year'] = data['date'].dt.year
    data['month'] = data['date'].dt.month
    data['day'] = data['date'].dt.day
    

Mini summary: Extract year, month, day from dates.

Lesson 4: Filtering by Date

Definition: You can filter data based on dates, like selecting data from a specific year.

Why it is important: It helps you focus on a specific time period.

Simple explanation: You select rows where the date is in 2024.

Real-life example: A company analyzes sales for the last quarter.

School example: A teacher looks at grades for the second semester.

Home example: You look at expenses for a specific month.

Nigerian example: A business analyzes transactions in a specific year.

Illustration:

    data_2024 = data[data['date'].dt.year == 2024]
    

Mini summary: Filter data by date.

Lesson 5: What is Merging?

Definition: Merging combines two datasets into one using a common column.

Why it is important: Often, data is stored in separate tables. Merging brings them together.

Simple explanation: You have a student list and a score list. You merge them on student ID.

Real-life example: A company merges customer data with sales data.

School example: A teacher merges student names with grades.

Home example: You merge your budget with your expenses.

Nigerian example: A business merges inventory data with sales data.

Illustration:

    merged = pd.merge(data1, data2, on='key_column')
    

Mini summary: Merging combines datasets.

Lesson 6: Types of Merges – Inner, Outer, Left, Right

Definition: There are different types of merges: inner (only matching rows), outer (all rows), left (all from left), right (all from right).

Why it is important: You choose the type based on what data you want to keep.

Simple explanation: Inner merge keeps only rows that appear in both datasets.

Real-life example: You use inner merge to find customers who have made a purchase.

School example: You use left merge to keep all students, even those without scores.

Home example: You use outer merge to combine all data.

Nigerian example: A business uses left merge to keep all inventory items.

Illustration:

    inner = pd.merge(data1, data2, on='key', how='inner')
    left = pd.merge(data1, data2, on='key', how='left')
    right = pd.merge(data1, data2, on='key', how='right')
    outer = pd.merge(data1, data2, on='key', how='outer')
    

Mini summary: Choose the type of merge based on your needs.

Lesson 7: Joining on Multiple Columns

Definition: Sometimes you need to merge on more than one column.

Why it is important: A single key may not be unique enough.

Simple explanation: You merge on both 'name' and 'city' to ensure a match.

Real-life example: A company merges on customer ID and location.

School example: A teacher merges on student name and class.

Home example: You merge on item name and store.

Nigerian example: A business merges on product code and region.

Illustration:

    merged = pd.merge(data1, data2, on=['name', 'city'])
    

Mini summary: Merge on multiple columns.

Lesson 8: Concatenating Data

Definition: Concatenating means stacking datasets on top of each other (rows) or side by side (columns).

Why it is important: You often need to combine data with the same columns.

Simple explanation: You have two datasets with the same columns. You stack them using pd.concat().

Real-life example: A company combines monthly sales data.

School example: A teacher combines grades from different classes.

Home example: You combine expenses from different months.

Nigerian example: A business combines branch sales data.

Illustration:

    combined = pd.concat([data1, data2])
    

Mini summary: Concatenating stacks data.

Lesson 9: What is Reshaping?

Definition: Reshaping changes how data is organized—rows to columns or columns to rows.

Why it is important: Reshaping makes data easier to analyze.

Simple explanation: You have data where each subject is a row. You reshape to make each subject a column.

Real-life example: A company reshapes sales data for better reporting.

School example: A teacher reshapes test scores by subject.

Home example: You reshape your budget by category.

Nigerian example: A business reshapes sales by region.

Illustration:

    Original: rows for each subject.
    Reshaped: columns for each subject.
    

Mini summary: Reshaping changes the layout of data.

Lesson 10: Pivot Tables

Definition: A pivot table summarizes data and allows you to view it in a new way.

Why it is important: It helps you analyze and summarize data quickly.

Simple explanation: You use pivot_table() to create a summary table.

Real-life example: A company creates a pivot table of sales by region.

School example: A teacher creates a pivot table of grades by subject.

Home example: You create a pivot table of expenses by category.

Nigerian example: A business creates a pivot table of sales by product.

Illustration:

    pivot = data.pivot_table(index='city', columns='product', values='sales', aggfunc='sum')
    

Mini summary: Pivot tables summarize data.

Lesson 11: Melting Data

Definition: Melting turns columns into rows. It is the reverse of pivoting.

Why it is important: Sometimes you need data in a long format for analysis.

Simple explanation: You use melt() to convert wide data to long data.

Real-life example: A company melts sales data for time series analysis.

School example: A teacher melts test scores for analysis.

Home example: You melt your budget for better tracking.

Nigerian example: A business melts sales data by quarter.

Illustration:

    melted = data.melt(id_vars=['name'], value_vars=['math', 'science'])
    

Mini summary: Melting turns columns into rows.

Lesson 12: Stacking and Unstacking

Definition: Stacking compresses columns into rows. Unstacking does the opposite.

Why it is important: It gives you more control over the layout.

Simple explanation: You use stack() to make data longer, and unstack() to make it wider.

Real-life example: A company stacks data for analysis.

School example: A teacher stacks scores for each student.

Home example: You stack your expenses for tracking.

Nigerian example: A business stacks sales by product.

Illustration:

    stacked = data.stack()
    unstacked = data.unstack()
    

Mini summary: Stacking and unstacking change data layout.

Lesson 13: Groupby Revisited

Definition: groupby() groups data and allows you to apply functions like sum, mean, or count.

Why it is important: It's essential for summarizing data.

Simple explanation: You group by city and calculate the average sales.

Real-life example: A company groups sales by product.

School example: A teacher groups grades by class.

Home example: You group expenses by category.

Nigerian example: A business groups sales by region.

Illustration:

    data.groupby('city')['sales'].mean()
    

Mini summary: groupby() summarizes data by category.

Lesson 14: Practice with Dates, Merging, and Reshaping

Definition: Practice applying all the techniques you learned.

Why it is important: Practice helps you remember and build skills.

Simple explanation: You take a messy dataset and clean dates, merge datasets, and reshape data.

Real-life example: A data analyst cleans a customer dataset.

School example: A student works on a class exercise.

Home example: You clean your own data.

Nigerian example: A Nigerian student uses local data.

Illustration:

    Convert dates: pd.to_datetime()
    Merge: pd.merge()
    Reshape: pivot_table() or melt()
    

Mini summary: Practice all three techniques.

Lesson 15: Your Advanced Data Cleaning Project

Definition: A project where you apply all advanced techniques to a real dataset.

Why it is important: It gives you real-world experience.

Simple explanation: You find a messy dataset with dates, multiple tables, and messy layout. You clean it completely.

Real-life example: A data analyst cleans a company dataset.

School example: A student completes a class project.

Home example: You clean your own expense data.

Nigerian example: A student uses a Nigerian dataset.

Illustration:

    Step 1: Load data.
    Step 2: Clean dates.
    Step 3: Merge datasets.
    Step 4: Reshape data.
    Step 5: Validate the cleaned data.
    

Mini summary: Apply advanced techniques to a real dataset.

🔑 Key Vocabulary

• to_datetime(): Converts dates to datetime format.
• Merge: Combines two datasets.
• Inner Merge: Keeps only matching rows.
• Outer Merge: Keeps all rows.
• Left Merge: Keeps all rows from left dataset.
• Right Merge: Keeps all rows from right dataset.
• Concatenate: Stacks data.
• Pivot Table: Summarizes data.
• Melt: Turns columns into rows.
• Stack: Compresses columns into rows.
• Unstack: Expands rows into columns.
• groupby(): Groups data for summary.

💡 Important Concepts

• Dates must be converted to datetime.
• Merging combines datasets.
• Choose the right type of merge.
• Pivot tables summarize data.
• Melting turns columns into rows.
• Stacking and unstacking change layout.
• groupby() summarizes by category.

📝 Step-by-Step Explanations

How to Clean Dates, Merge, and Reshape Data in 6 Steps:

1. Load Data: data = pd.read_csv("file.csv")
2. Clean Dates: data['date'] = pd.to_datetime(data['date'])
3. Merge Data: merged = pd.merge(data1, data2, on='key')
4. Pivot Data: pivot = data.pivot_table(index='col1', columns='col2', values='value')
5. Melt Data: melted = data.melt(id_vars=['id'], value_vars=['col1', 'col2'])
6. Validate: Check the cleaned data.

🌍 Real-life Examples

• A company merges customer data with sales data.
• A school converts date formats for attendance.
• A hospital reshapes patient data for analysis.
• A business uses pivot tables for sales reports.

🇳🇬 Nigerian Examples

• A Nigerian bank merges account data with transaction data.
• A Nigerian school converts date formats for student records.
• A Nigerian business uses pivot tables to analyze sales by region.
• A Nigerian hospital reshapes patient data for research.

🎈 Fun Examples Children Relate To

• Merging a list of friends with their favorite colors.
• Converting dates of birthdays.
• Pivoting a table of sports scores.
• Melting a list of books and authors.

🏠 Everyday Examples

• Merging your budget with your expenses.
• Converting dates in your calendar.
• Pivoting your grocery list by category.
• Melting your to-do list by priority.

🧑‍🏫 Teacher Notes

This module covers advanced data cleaning topics. Teachers should emphasize the importance of dates and merging in real-world scenarios. Use examples with multiple datasets. Show students how to use pivot tables and melting effectively. Encourage practice with real datasets.

👪 Parent Tips

Parents can help children understand these concepts by using everyday examples like merging a shopping list with prices or converting dates on a calendar. Encourage children to think about how data is organized and how it can be reshaped.

🌟 Interesting Facts

• Dates are often the most challenging part of data cleaning.
• Merging is one of the most common operations in data analysis.
• Pivot tables were invented by Microsoft Excel and are now used in Pandas.
• Reshaping data is essential for many machine learning algorithms.

🤔 Did You Know?

• Did you know that dates in different formats can cause errors in analysis?
• Did you know that merging is like a VLOOKUP in Excel?
• Did you know that pivot tables are used in almost every industry?
• Did you know that melting data is often needed for data visualization?

🧠 Remember This

• Convert dates with pd.to_datetime().
• Merge data with pd.merge().
• Choose inner, outer, left, or right merge.
• Pivot tables summarize data.
• Melting turns columns into rows.
• Stacking and unstacking change layout.
• groupby() summarizes by category.

❌ Common Mistakes

• Not converting dates: Dates stay as strings.
• Using the wrong merge type: Losing important data.
• Not specifying the merge key: Error in merging.
• Using pivot incorrectly: Wrong index or columns.
• Forgetting to validate: Not checking the cleaned data.

✅ Best Practices

• Always convert dates to datetime.
• Choose the right merge type.
• Specify the merge key clearly.
• Use pivot tables for summary.
• Use melt for long format data.
• Validate after cleaning.
• Document your steps.

📊 Clear ASCII Illustrations

Merging Flowchart

    Data1 + Data2 -> Merge -> Combined Data
    

Pivot Table Illustration

    Original:
    City    Product   Sales
    Lagos   Rice      100
    Abuja   Rice      200
    Lagos   Beans     150
    Abuja   Beans     250

    Pivot:
    City    Rice    Beans
    Lagos   100     150
    Abuja   200     250
    

📋 Comparison Tables

Merge Types

Reshaping Methods

📌 End-of-Module Summary

In Module Five, we learned advanced data cleaning skills: working with dates, merging datasets, and reshaping data. We converted dates to datetime format, extracted date components, and filtered by date. We merged datasets using inner, outer, left, and right joins. We also learned how to concatenate data. For reshaping, we used pivot tables, melting, stacking, and unstacking. We also revisited groupby() for summary analysis. These skills are essential for handling complex real-world datasets. You are now a true data cleaning expert!

❓ Frequently Asked Questions

1. How do you convert dates? Use pd.to_datetime().
2. What is merging? Combining two datasets.
3. What is an inner merge? Keeps only matching rows.
4. What is a pivot table? A summary table.
5. What does melt do? Turns columns into rows.
6. What does stack do? Compresses columns.
7. What does unstack do? Expands rows.
8. What is groupby() used for? Summarizing data by category.
9. Why do we reshape data? To make it easier to analyze.
10. What is the first step in working with dates? Convert to datetime.

📝 Review Questions

1. How do you convert dates in Pandas?
2. What is merging?
3. What is an inner merge?
4. What is a left merge?
5. What is a pivot table?
6. What does melt do?
7. What does stack do?
8. What does unstack do?
9. What is groupby() used for?
10. Why do we reshape data?
11. Give a Nigerian example of merging.
12. Give a Nigerian example of a pivot table.
13. What is the first step in working with dates?
14. What is the difference between melt and stack?
15. Why is it important to validate after cleaning?

✍️ Fill-in-the-Blank

1. Use ______ to convert dates. (pd.to_datetime())
2. ______ combines two datasets. (Merging)
3. An ______ merge keeps only matching rows. (inner)
4. A ______ merge keeps all rows from the left dataset. (left)
5. A ______ summarizes data. (pivot table)
6. ______ turns columns into rows. (Melt)
7. ______ compresses columns into rows. (Stack)
8. ______ expands rows into columns. (Unstack)
9. ______ groups data for summary. (groupby())
10. You should always ______ after cleaning. (validate)

✅ True or False

1. pd.to_datetime() converts dates. (True)
2. Merging combines datasets. (True)
3. An inner merge keeps all rows. (False)
4. A left merge keeps all rows from the left dataset. (True)
5. A pivot table is a chart. (False)
6. Melting turns rows into columns. (False)
7. Stacking compresses columns. (True)
8. Unstacking expands rows. (True)
9. groupby() summarizes data. (True)
10. You don't need to validate after cleaning. (False)

🔘 Multiple Choice Questions

1. How do you convert dates?
   A) str.date()
   B) pd.to_datetime()
   C) convert.date()
   D) date_format()
   Answer: B
2. What is merging?
   A) Combining data
   B) Removing data
   C) Converting data
   D) Reshaping data
   Answer: A
3. What is an inner merge?
   A) Keeps all rows
   B) Keeps only matching rows
   C) Keeps left rows
   D) Keeps right rows
   Answer: B
4. What is a pivot table?
   A) A chart
   B) A summary table
   C) A type of merge
   D) A date format
   Answer: B
5. What does melt do?
   A) Turns columns into rows
   B) Turns rows into columns
   C) Converts dates
   D) Removes duplicates
   Answer: A
6. What does stack do?
   A) Compresses columns
   B) Expands rows
   C) Converts dates
   D) Merges data
   Answer: A
7. What does unstack do?
   A) Compresses columns
   B) Expands rows
   C) Converts dates
   D) Merges data
   Answer: B
8. What is groupby() used for?
   A) Merging data
   B) Summarizing data by category
   C) Converting dates
   D) Reshaping data
   Answer: B
9. What is the first step in working with dates?
   A) Extract year
   B) Convert to datetime
   C) Filter
   D) Group
   Answer: B
10. What is a left merge?
    A) Keeps all rows from left
    B) Keeps all rows from right
    C) Keeps only matching
    D) Keeps all rows
    Answer: A
11. What does pd.concat() do?
    A) Merges data
    B) Concatenates data
    C) Converts dates
    D) Reshapes data
    Answer: B
12. Which is a Nigerian example of merging?
    A) Merging student names with grades
    B) Merging product names with sales
    C) Merging customer data with transactions
    D) All of the above
    Answer: D
13. Why reshape data?
    A) To make it easier to analyze
    B) To make it harder
    C) To delete data
    D) To merge data
    Answer: A
14. What is the difference between melt and stack?
    A) Melt turns columns into rows, stack compresses columns
    B) Stack turns columns into rows, melt compresses columns
    C) They are the same
    D) They are both used for dates
    Answer: A
15. Why validate after cleaning?
    A) To make sure it's correct
    B) To make it harder
    C) To delete data
    D) To merge data
    Answer: A

🔗 Matching Exercises

Match the term with its description:

Answers: 1-C, 2-A, 3-B, 4-D, 5-E

✏️ Short Answer Questions

1. How do you convert dates in Pandas?
2. What is merging and why is it important?
3. What is the difference between an inner and left merge?
4. What does melt do and when would you use it?
5. Give a Nigerian example of using a pivot table.

🎭 Scenario-based Exercises

Scenario 1: You have two datasets—student names and test scores. Merge them using student ID.

Scenario 2: You have a dataset with dates in multiple formats. Convert them to a single standard format.

Scenario 3: You have sales data with city, product, and sales. Create a pivot table showing sales by city and product.

👥 Group Activity

In groups, get two related datasets. Merge them using the appropriate merge type. Then, reshape the merged data using a pivot table. Present your cleaned and reshaped data to the class.

🧑 Individual Activity

Create a dataset with dates, merge it with another dataset, and reshape it using melt. Write Python code to perform all steps. Show your results.

💬 Classroom Discussion Questions

• Why are dates often messy?
• When would you use an outer merge?
• How can pivot tables help businesses?
• What is the advantage of melting data?
• How can these skills be used in Nigeria?

🛠️ Mini Project

Create a "Data Cleaning Report" for a dataset that includes dates, multiple tables, and messy layout. Clean the dates, merge the tables, and reshape the data. Show your methods and the final cleaned dataset.

📄 Practical Assignment

Download a dataset with dates and merge it with another dataset. Clean the dates, merge the data, and reshape it. Write a report explaining your steps and show the final dataset.

🏆 Challenge Exercise

Find a dataset with multiple tables and messy dates. Write a Python script that cleans the dates, merges the tables, and reshapes the data. Test it on the dataset and show the results.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Convert dates with pd.to_datetime().
• Merge data with pd.merge().
• Choose inner, outer, left, or right merge.
• Pivot tables summarize data.
• Melting turns columns into rows.
• Stacking and unstacking change layout.
• groupby() summarizes by category.
• Always validate after cleaning.

🔜 Preparation for Module Six

In Module Six, we will learn about automating data cleaning with functions and pipelines. You will learn how to write reusable code to clean any dataset. Get ready to become a data cleaning automation expert!

End of Module Five – You are now an advanced data cleaner!

📘 Module Six: Python for Data Cleaning – Automation with Functions and Pipelines

Welcome to Module Six! In all the previous modules, we learned how to clean data step by step. But doing the same steps over and over for every dataset can be tiring. That's where automation comes in. We can write functions and create pipelines that automatically clean data for us. This saves time and reduces errors. By the end of this module, you will be able to write reusable code to clean any dataset. Let's begin!

🎯 Learning Objectives

After this module, you will be able to:

• Write functions to automate data cleaning tasks.
• Create a data cleaning pipeline.
• Apply functions and pipelines to datasets.
• Validate and test your automated cleaning code.

📖 Warm-up Story: The Automated Cleaning System

A company in Nigeria received new data every week. It always had the same problems—missing values, duplicates, and inconsistent text. A data analyst used to clean it manually every week. It took hours.

Then she wrote a Python function that did all the cleaning steps automatically. She ran the function on the new data each week. In minutes, the data was clean. She saved hours every week and never made mistakes.

This module teaches you how to write your own automated cleaning functions and pipelines.

📚 Main Lessons

Lesson 1: Why Automate Data Cleaning?

Definition: Automation means using code to do repetitive tasks without human intervention.

Why it is important: It saves time, reduces errors, and makes your work consistent.

Simple explanation: Like a machine that washes dishes, automation does the work for you.

Real-life example: A company automates data cleaning for weekly reports.

School example: A student automates cleaning of class attendance data.

Home example: You automate sorting your expenses.

Nigerian example: A Nigerian business automates customer data cleaning.

Illustration:

    Manual Cleaning: 2 hours
    Automated Cleaning: 2 minutes
    

Mini summary: Automation saves time and reduces errors.

Lesson 2: What is a Function?

Definition: A function is a block of code that performs a specific task. You can reuse it whenever you need it.

Why it is important: Functions make your code organized and reusable.

Simple explanation: Like a recipe for making pancakes. You can use the same recipe every time.

Real-life example: A function that cleans missing values.

School example: A function that calculates average grades.

Home example: A function that organizes your to-do list.

Nigerian example: A function that standardizes Nigerian city names.

Illustration:

    def clean_missing(data):
        data = data.fillna(0)
        return data
    

Mini summary: Functions are reusable blocks of code.

Lesson 3: Writing a Simple Cleaning Function

Definition: A cleaning function takes a dataset, applies cleaning steps, and returns the cleaned dataset.

Why it is important: It packages all your cleaning steps into one easy-to-use tool.

Simple explanation: You write a function that does all the cleaning steps you learned in previous modules.

Real-life example: A function that removes duplicates and fills missing values.

School example: A function that cleans student grades.

Home example: A function that cleans your budget.

Nigerian example: A function that cleans sales data.

Illustration:

    def clean_data(data):
        data = data.drop_duplicates()
        data = data.fillna(0)
        return data
    

Mini summary: A cleaning function applies all cleaning steps.

Lesson 4: Adding More Steps to Your Function

Definition: You can add multiple cleaning steps to your function.

Why it is important: One function can do all the cleaning work.

Simple explanation: You add steps for duplicates, missing values, text standardization, and outliers.

Real-life example: A function that removes duplicates, fills missing values, and standardizes text.

School example: A function that cleans attendance data.

Home example: A function that cleans your expense data.

Nigerian example: A function that cleans customer data.

Illustration:

    def clean_data(data):
        data = data.drop_duplicates()
        data = data.fillna(0)
        data['name'] = data['name'].str.lower().str.strip()
        return data
    

Mini summary: Add all cleaning steps to one function.

Lesson 5: Using Parameters in Functions

Definition: Parameters are variables you pass to a function to make it more flexible.

Why it is important: You can customize the cleaning for different datasets.

Simple explanation: You can tell the function what value to fill missing data with.

Real-life example: A function that lets you choose the fill value.

School example: A function that lets you choose the columns to clean.

Home example: A function that lets you choose the date format.

Nigerian example: A function that lets you choose which city to standardize.

Illustration:

    def clean_data(data, fill_value=0):
        data = data.drop_duplicates()
        data = data.fillna(fill_value)
        return data
    

Mini summary: Parameters make functions flexible.

Lesson 6: What is a Pipeline?

Definition: A pipeline is a sequence of steps that data goes through. In Pandas, you can use the pipe() function to create pipelines.

Why it is important: Pipelines make your code clean and organized.

Simple explanation: Think of a pipeline as a conveyor belt. Data goes in one end, and clean data comes out the other.

Real-life example: A data pipeline that loads, cleans, and saves data.

School example: A pipeline that cleans and analyzes student data.

Home example: A pipeline that cleans and sorts your expenses.

Nigerian example: A pipeline that processes sales data.

Illustration:

    Data -> Step 1 -> Step 2 -> Step 3 -> Clean Data
    

Mini summary: Pipelines organize data cleaning steps.

Lesson 7: Creating a Pipeline with pipe()

Definition: The pipe() function allows you to chain multiple functions together.

Why it is important: It makes your code easier to read and maintain.

Simple explanation: You use pipe() to apply one function after another.

Real-life example: A pipeline that applies cleaning functions in order.

School example: A pipeline that cleans grades, then calculates averages.

Home example: A pipeline that cleans and sorts expenses.

Nigerian example: A pipeline that cleans and summarizes sales.

Illustration:

    data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
    

Mini summary: pipe() chains cleaning functions.

Lesson 8: Writing Pipeline Functions

Definition: Pipeline functions are functions designed to be used in a pipe. They take data as input and return cleaned data.

Why it is important: They are modular and reusable.

Simple explanation: Each function does one cleaning task. You can put them together in any order.

Real-life example: A function that drops duplicates, a function that fills missing values.

School example: A function that standardizes names.

Home example: A function that removes outliers.

Nigerian example: A function that standardizes city names.

Illustration:

    def remove_duplicates(data):
        return data.drop_duplicates()

    def fill_missing(data):
        return data.fillna(0)

    def standardize_text(data):
        data['name'] = data['name'].str.lower().str.strip()
        return data
    

Mini summary: Pipeline functions are modular cleaning steps.

Lesson 9: Building a Complete Cleaning Pipeline

Definition: A complete cleaning pipeline combines all functions in the right order.

Why it is important: It automatically cleans data from start to finish.

Simple explanation: You chain all your cleaning functions using pipe().

Real-life example: A full pipeline for cleaning customer data.

School example: A full pipeline for cleaning attendance data.

Home example: A full pipeline for cleaning expenses.

Nigerian example: A full pipeline for cleaning sales data.

Illustration:

    pipeline = data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
    clean_data = pipeline
    

Mini summary: A complete pipeline cleans data automatically.

Lesson 10: Testing Your Pipeline

Definition: Testing means checking if your pipeline works correctly.

Why it is important: You need to make sure your pipeline cleans data properly.

Simple explanation: You run your pipeline on a small dataset and check the output.

Real-life example: A data analyst tests a pipeline on sample data.

School example: A student tests a pipeline on a small dataset.

Home example: You test your pipeline on a few days of expenses.

Nigerian example: A business tests a pipeline on a sample sales file.

Illustration:

    test_data = pd.read_csv("test.csv")
    cleaned = test_data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
    print(cleaned.head())
    

Mini summary: Test your pipeline to ensure it works.

Lesson 11: Saving and Reusing Pipelines

Definition: You can save your pipeline as a Python script or function to reuse it.

Why it is important: You can use the same pipeline on new datasets.

Simple explanation: You write your pipeline in a file and call it whenever you need to clean data.

Real-life example: A company saves a cleaning pipeline for weekly reports.

School example: A student saves a pipeline for class projects.

Home example: You save a pipeline to clean your monthly budget.

Nigerian example: A business saves a pipeline for quarterly sales data.

Illustration:

    def cleaning_pipeline(data):
        return data.pipe(remove_duplicates).pipe(fill_missing).pipe(standardize_text)
    

Mini summary: Save pipelines to reuse them.

Lesson 12: Error Handling in Pipelines

Definition: Error handling means managing errors that might occur during cleaning.

Why it is important: Sometimes data is unexpected. You need to handle errors gracefully.

Simple explanation: You use try-except blocks to catch errors and continue.

Real-life example: A pipeline that handles missing columns.

School example: A pipeline that handles missing data.

Home example: A pipeline that handles missing files.

Nigerian example: A pipeline that handles missing fields.

Illustration:

    def safe_clean(data):
        try:
            return data.pipe(remove_duplicates)
        except Exception as e:
            print("Error:", e)
            return data
    

Mini summary: Handle errors to make pipelines robust.

Lesson 13: Scheduling Automated Cleaning

Definition: Scheduling means running your pipeline automatically at set times.

Why it is important: It ensures data is cleaned regularly without manual effort.

Simple explanation: You use a scheduler to run your Python script every day or week.

Real-life example: A company schedules a pipeline to run every Sunday night.

School example: A teacher schedules a pipeline to run after each test.

Home example: You schedule a pipeline to run every month.

Nigerian example: A business schedules a pipeline for monthly reports.

Illustration:

    # Example: Using a scheduler (like cron or Task Scheduler)
    # Not code, just concept.
    

Mini summary: Schedule pipelines for regular cleaning.

Lesson 14: Practice Building a Pipeline

Definition: Practice building a pipeline from scratch.

Why it is important: Practice makes you comfortable with automation.

Simple explanation: You take a messy dataset and build a pipeline to clean it.

Real-life example: A data analyst builds a pipeline for a new client.

School example: A student builds a pipeline for a class project.

Home example: You build a pipeline for your personal budget.

Nigerian example: A Nigerian student builds a pipeline for local data.

Illustration:

    Step 1: Define cleaning functions.
    Step 2: Chain them with pipe().
    Step 3: Test on a dataset.
    Step 4: Use on new data.
    

Mini summary: Practice building pipelines.

Lesson 15: Your Automation Project

Definition: An automation project where you build a pipeline for a real dataset.

Why it is important: It gives you real-world experience.

Simple explanation: You find a messy dataset and build an automated cleaning pipeline.

Real-life example: A data analyst builds a pipeline for a company.

School example: A student completes a class project.

Home example: You build a pipeline for your own data.

Nigerian example: A student uses a Nigerian dataset.

Illustration:

    Step 1: Load data.
    Step 2: Write cleaning functions.
    Step 3: Build pipeline.
    Step 4: Test and validate.
    Step 5: Save and reuse.
    

Mini summary: Build an automated cleaning project.

🔑 Key Vocabulary

• Automation: Using code to do tasks automatically.
• Function: A reusable block of code.
• Pipeline: A sequence of steps in a process.
• pipe(): A Pandas function to chain steps.
• Parameter: A variable passed to a function.
• Error Handling: Managing errors gracefully.
• Scheduling: Running code at set times.
• Validation: Checking if data is clean.

💡 Important Concepts

• Automation saves time.
• Functions are reusable.
• Pipelines organize cleaning steps.
• Parameters make functions flexible.
• Test pipelines on sample data.
• Handle errors to avoid crashes.
• Schedule pipelines for regular use.

📝 Step-by-Step Explanations

How to Build a Cleaning Pipeline in 6 Steps:

1. Load Data: data = pd.read_csv("file.csv")
2. Define Functions: Write functions for each cleaning task.
3. Chain Functions: Use pipe() to chain them.
4. Test Pipeline: Run on sample data.
5. Validate: Check cleaned data.
6. Save and Reuse: Save as a script or function.

🌍 Real-life Examples

• A company automates cleaning of customer data.
• A school automates cleaning of student records.
• A hospital automates cleaning of patient data.
• A bank automates cleaning of transaction data.

🇳🇬 Nigerian Examples

• A Nigerian business automates cleaning of sales data.
• A Nigerian school automates cleaning of attendance data.
• A Nigerian hospital automates cleaning of patient records.
• A Nigerian bank automates cleaning of customer data.

🎈 Fun Examples Children Relate To

• A function that cleans a list of video games.
• A pipeline that cleans your favorite movies.
• A function that organizes your toys.
• A pipeline that cleans your school schedule.

🏠 Everyday Examples

• A function that cleans your to-do list.
• A pipeline that organizes your expenses.
• A function that standardizes your contact list.
• A pipeline that cleans your calendar.

🧑‍🏫 Teacher Notes

This module focuses on automation and efficiency. Teachers should emphasize the importance of writing reusable code. Use examples of functions and pipelines. Encourage students to build their own pipelines for different datasets. Highlight the benefits of automation for real-world data cleaning.

👪 Parent Tips

Parents can help children understand automation by showing them how they can automate simple tasks, like sorting emails or organizing files. Encourage children to think about how they can use code to make their daily tasks easier.

🌟 Interesting Facts

• Automation is used in almost every industry.
• Functions are a core concept in programming.
• Pipelines are used in many data science workflows.
• Automation can save businesses millions of hours.

🤔 Did You Know?

• Did you know that automation is a key part of "DevOps"?
• Did you know that functions can be reused in many projects?
• Did you know that pipelines are used in machine learning?
• Did you know that many Nigerian companies are adopting automation?

🧠 Remember This

• Automation saves time and reduces errors.
• Functions are reusable blocks of code.
• Pipelines organize cleaning steps.
• Parameters make functions flexible.
• Test pipelines before using them.
• Handle errors to make pipelines robust.
• Schedule pipelines for regular cleaning.

❌ Common Mistakes

• Not testing the pipeline: Using it on data without testing.
• Forgetting parameters: Not making functions flexible.
• Not handling errors: Letting errors crash the pipeline.
• Not validating: Not checking the cleaned data.
• Overcomplicating: Making functions too complex.

✅ Best Practices

• Write functions for each cleaning task.
• Use parameters to make functions flexible.
• Chain functions with pipe().
• Test pipelines on sample data.
• Validate cleaned data.
• Handle errors gracefully.
• Save and reuse pipelines.

📊 Clear ASCII Illustrations

Pipeline Flowchart

    Data -> Remove Duplicates -> Fill Missing -> Standardize Text -> Clean Data
    

Function Diagram

    +-------------------+
    |  Function         |
    |  Input -> Step -> |
    |  Output           |
    +-------------------+
    

📋 Comparison Tables

Manual vs Automated Cleaning

Function vs Pipeline

📌 End-of-Module Summary

In Module Six, we learned how to automate data cleaning using functions and pipelines. We wrote functions to perform cleaning tasks and chained them together using pipe(). We added parameters to make functions flexible. We also learned how to test pipelines, handle errors, and schedule them for regular use. Automation saves time, reduces errors, and makes your workflow efficient. You are now ready to build your own automated data cleaning pipelines!

❓ Frequently Asked Questions

1. Why automate data cleaning? Saves time and reduces errors.
2. What is a function? A reusable block of code.
3. What is a pipeline? A sequence of cleaning steps.
4. What does pipe() do? Chains functions together.
5. What are parameters? Variables passed to a function.
6. How do you test a pipeline? Run it on sample data.
7. What is error handling? Managing errors gracefully.
8. How do you schedule a pipeline? Use a scheduler like cron.
9. What is validation? Checking cleaned data.
10. What is the first step in building a pipeline? Define cleaning functions.

📝 Review Questions

1. Why automate data cleaning?
2. What is a function?
3. What is a pipeline?
4. What does pipe() do?
5. What are parameters?
6. How do you test a pipeline?
7. What is error handling?
8. How do you schedule a pipeline?
9. What is validation?
10. What is the first step in building a pipeline?
11. Give a Nigerian example of automation.
12. Give a home example of a pipeline.
13. What is the benefit of using functions?
14. What is the benefit of using pipelines?
15. Why is testing important?

✍️ Fill-in-the-Blank

1. ______ saves time and reduces errors. (Automation)
2. A ______ is a reusable block of code. (function)
3. A ______ is a sequence of cleaning steps. (pipeline)
4. Use ______ to chain functions. (pipe())
5. ______ are variables passed to a function. (Parameters)
6. Test a pipeline on ______ data. (sample)
7. ______ handling manages errors gracefully. (Error)
8. You can ______ a pipeline to run at set times. (schedule)
9. ______ means checking the cleaned data. (Validation)
10. The first step in building a pipeline is to ______ cleaning functions. (define)

✅ True or False

1. Automation saves time. (True)
2. A function is a reusable block of code. (True)
3. A pipeline is a single step. (False)
4. pipe() chains functions together. (True)
5. Parameters make functions less flexible. (False)
6. You should test pipelines on sample data. (True)
7. Error handling is not important. (False)
8. You can schedule pipelines to run automatically. (True)
9. Validation is optional. (False)
10. Defining functions is the first step in building a pipeline. (True)

🔘 Multiple Choice Questions

1. Why automate data cleaning?
   A) Saves time
   B) Reduces errors
   C) Both A and B
   D) Neither
   Answer: C
2. What is a function?
   A) A reusable block of code
   B) A single line of code
   C) A type of data
   D) A file
   Answer: A
3. What is a pipeline?
   A) A single step
   B) A sequence of steps
   C) A type of function
   D) A file format
   Answer: B
4. What does pipe() do?
   A) Chains functions
   B) Removes duplicates
   C) Fills missing values
   D) Saves data
   Answer: A
5. What are parameters?
   A) Variables passed to a function
   B) A type of data
   C) A file
   D) A function
   Answer: A
6. How do you test a pipeline?
   A) Run on sample data
   B) Run on full data
   C) Guess
   D) Ignore
   Answer: A
7. What is error handling?
   A) Managing errors
   B) Ignoring errors
   C) Creating errors
   D) Deleting errors
   Answer: A
8. How do you schedule a pipeline?
   A) Using a scheduler
   B) Manually
   C) Ignoring it
   D) By guess
   Answer: A
9. What is validation?
   A) Checking cleaned data
   B) Deleting cleaned data
   C) Ignoring cleaned data
   D) Creating cleaned data
   Answer: A
10. What is the first step in building a pipeline?
    A) Define functions
    B) Load data
    C) Save data
    D) Test data
    Answer: A
11. Which is a Nigerian example of automation?
    A) Cleaning sales data automatically
    B) Cleaning data manually
    C) Ignoring data
    D) Deleting data
    Answer: A
12. What is the benefit of using functions?
    A) Reusability
    B) Speed
    C) Complexity
    D) Confusion
    Answer: A
13. What is the benefit of using pipelines?
    A) Organization
    B) Messiness
    C) Confusion
    D) Slowness
    Answer: A
14. Why is testing important?
    A) To ensure correctness
    B) To create errors
    C) To waste time
    D) To confuse users
    Answer: A
15. What is the main goal of automation?
    A) To save time and reduce errors
    B) To create more work
    C) To make things complicated
    D) To confuse people
    Answer: A

🔗 Matching Exercises

Match the term with its description:

Answers: 1-B, 2-A, 3-C, 4-D, 5-E

✏️ Short Answer Questions

1. Why is automation important in data cleaning?
2. What is a function and why is it useful?
3. What is a pipeline and how does it help?
4. How do you test a cleaning pipeline?
5. Give a Nigerian example of using automation.

🎭 Scenario-based Exercises

Scenario 1: You receive a new dataset every week. The data always has duplicates and missing values. How would you automate the cleaning process?

Scenario 2: You have a dataset with inconsistent city names and extra spaces. Write a function to clean it.

Scenario 3: You want to build a pipeline that removes duplicates, fills missing values, and standardizes text. How would you do it?

👥 Group Activity

In groups, build a cleaning pipeline for a dataset of your choice. Each group member writes one cleaning function. Combine them into a pipeline and test it. Present your pipeline to the class.

🧑 Individual Activity

Write a function that cleans a dataset by removing duplicates, filling missing values, and standardizing text. Test it on a sample dataset. Write a short report on your results.

💬 Classroom Discussion Questions

• Why is automation important for businesses?
• What are some challenges in building pipelines?
• How can automation help in Nigerian schools?
• What is the role of testing in automation?
• How can you make pipelines more robust?

🛠️ Mini Project

Create an "Automated Data Cleaning Pipeline" for a dataset of your choice. Include functions for duplicates, missing values, text standardization, and outliers. Test it on a sample dataset and save the pipeline for reuse.

📄 Practical Assignment

Find a messy dataset. Build a cleaning pipeline for it. Write a report explaining your pipeline, including the functions and the cleaning steps. Show the cleaned dataset.

🏆 Challenge Exercise

Build a pipeline that automatically cleans a new dataset every week. Use scheduling to run it at a set time. Write a script that loads, cleans, and saves the data automatically.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Automation saves time and reduces errors.
• Functions are reusable blocks of code.
• Pipelines organize cleaning steps.
• Parameters make functions flexible.
• Test pipelines on sample data.
• Handle errors to make pipelines robust.
• Schedule pipelines for regular cleaning.

🔜 Preparation for Module Seven

In Module Seven, we will learn how to validate and test data quality after cleaning. You will learn how to use asserts, data profiling, and quality checks. Get ready to ensure your data is perfect!

End of Module Six – You are now an automation expert!

📘 Module Seven: Python for Data Cleaning – Data Quality, Validation, and Testing

Welcome to Module Seven! In all the previous modules, we learned how to clean data—removing duplicates, fixing missing values, and more. But how do we know our data is really clean? How do we make sure we didn't make any mistakes? In this module, we will learn about data quality, validation, and testing. These are the final checks to make sure your data is perfect. By the end of this module, you will be able to test your data for quality and be confident in your cleaning. Let's begin!

🎯 Learning Objectives

After this module, you will be able to:

• Explain what data quality means.
• Use assertions to test your data.
• Create data quality checks.
• Validate data after cleaning.
• Apply these techniques to real datasets.

📖 Warm-up Story: The Trusted Report

A school in Nigeria needed to send a report to the government. The principal was nervous. What if the data had errors? What if some numbers were wrong? A data analyst said: "Don't worry. I will validate the data."

She used Python to check the data. She made sure there were no missing values. She checked that all ages were between 5 and 18. She verified that totals added up correctly. She even wrote tests to catch any mistakes. The report was perfect. The government trusted the data.

This module teaches you how to validate your data and build trust in your work.

📚 Main Lessons

Lesson 1: What is Data Quality?

Definition: Data quality means that data is accurate, complete, consistent, and reliable.

Why it is important: Low-quality data leads to wrong decisions. High-quality data leads to good decisions.

Simple explanation: Good data is like clean water—safe and ready to use. Bad data is like dirty water—you don't want to drink it!

Real-life example: A hospital uses high-quality patient data to give the right treatment.

School example: A teacher uses high-quality grades to know which students need help.

Home example: You use a clean budget to manage your money.

Nigerian example: A Nigerian government uses high-quality census data for planning.

Illustration:

    High Quality Data = Accurate + Complete + Consistent
    

Mini summary: Data quality means data is accurate and reliable.

Lesson 2: The Dimensions of Data Quality

Definition: Dimensions of data quality are the different aspects that make data good: accuracy, completeness, consistency, timeliness, and validity.

Why it is important: You need to check each dimension to ensure overall quality.

Simple explanation: Think of them as the 5 checks for good data.

Real-life example: A company checks that customer addresses are complete (completeness) and correct (accuracy).

School example: A teacher checks that all grades are entered (completeness) and within a valid range (validity).

Home example: You check that all expenses are recorded (completeness) and the amounts are correct (accuracy).

Nigerian example: A business checks that all sales are recorded (completeness) and prices are correct (accuracy).

Illustration:

    Dimensions:
    1. Accuracy
    2. Completeness
    3. Consistency
    4. Timeliness
    5. Validity
    

Mini summary: Data quality has five dimensions to check.

Lesson 3: What is Data Validation?

Definition: Data validation is the process of checking data to ensure it meets certain rules or standards.

Why it is important: Validation catches errors before you use the data.

Simple explanation: It's like checking your work before you submit it.

Real-life example: A bank validates that account numbers are correct.

School example: A teacher validates that student IDs are valid.

Home example: You validate that your budget totals add up.

Nigerian example: A business validates that product codes are correct.

Illustration:

    Data -> Validation Rules -> Valid Data or Error
    

Mini summary: Validation checks if data follows rules.

Lesson 4: Using Asserts in Python

Definition: An assert statement checks if a condition is true. If it's false, it raises an error.

Why it is important: Asserts help you catch errors early in your code.

Simple explanation: You say: "I assert that all ages are > 0." If any age is less than 0, Python will stop and tell you.

Real-life example: A data analyst uses asserts to check that no negative prices exist.

School example: A student uses asserts to check that grades are between 0 and 100.

Home example: You use asserts to check that expenses are positive.

Nigerian example: A business uses asserts to check that sales are positive.

Illustration:

    assert data['age'].min() > 0, "Age must be positive!"
    

Mini summary: Assertions check conditions and raise errors if they fail.

Lesson 5: Common Validation Checks

Definition: Common validation checks include checking data types, ranges, uniqueness, and format.

Why it is important: These are the most common ways data can be wrong.

Simple explanation: You check that numbers are numbers, dates are dates, and values are within expected limits.

Real-life example: A company checks that phone numbers have 10 digits.

School example: A teacher checks that student IDs are unique.

Home example: You check that dates are in the past.

Nigerian example: A business checks that transaction amounts are positive.

Illustration:

    Checks:
    - Data type check
    - Range check
    - Uniqueness check
    - Format check
    

Mini summary: Common validation checks catch typical errors.

Lesson 6: Validating Data Types

Definition: Validating data types means checking that columns have the correct data type (e.g., int, float, string).

Why it is important: Wrong data types can cause errors in analysis.

Simple explanation: You check that the "age" column contains integers, not text.

Real-life example: A company checks that the "price" column is a float.

School example: A teacher checks that "grade" is an integer.

Home example: You check that "amount" is a number.

Nigerian example: A business checks that "sales" is a float.

Illustration:

    assert data['age'].dtype == 'int', "Age must be integer!"
    

Mini summary: Validate that data types are correct.

Lesson 7: Validating Value Ranges

Definition: Validating value ranges means checking that values fall within an expected minimum and maximum.

Why it is important: Out-of-range values are often errors.

Simple explanation: You check that ages are between 0 and 120.

Real-life example: A hospital checks that patient ages are reasonable.

School example: A teacher checks that grades are between 0 and 100.

Home example: You check that expenses are positive.

Nigerian example: A business checks that salaries are within a reasonable range.

Illustration:

    assert data['age'].between(0, 120).all(), "Age out of range!"
    

Mini summary: Validate that values are within expected ranges.

Lesson 8: Validating Uniqueness

Definition: Validating uniqueness means checking that values in a column are unique (no duplicates).

Why it is important: Duplicate values can cause problems, especially for IDs.

Simple explanation: You check that student IDs are all different.

Real-life example: A company checks that customer emails are unique.

School example: A teacher checks that student IDs are unique.

Home example: You check that item names are unique.

Nigerian example: A business checks that product codes are unique.

Illustration:

    assert data['id'].is_unique, "IDs must be unique!"
    

Mini summary: Validate that key columns have unique values.

Lesson 9: Validating Format

Definition: Validating format means checking that values follow a specific pattern, like email or date formats.

Why it is important: Wrong formats can cause errors.

Simple explanation: You check that email addresses have an "@" symbol.

Real-life example: A company checks that email addresses are valid.

School example: A teacher checks that dates are in the correct format.

Home example: You check that phone numbers have 10 digits.

Nigerian example: A business checks that phone numbers are valid.

Illustration:

    assert data['email'].str.contains('@').all(), "Invalid email format!"
    

Mini summary: Validate that values follow the right format.

Lesson 10: Creating a Validation Function

Definition: A validation function is a function that checks multiple validation rules on a dataset.

Why it is important: It organizes all your validation checks in one place.

Simple explanation: You write a function that runs all your checks and reports any issues.

Real-life example: A data analyst writes a validation function for new data.

School example: A student writes a validation function for a project.

Home example: You write a validation function for your budget.

Nigerian example: A business writes a validation function for sales data.

Illustration:

    def validate_data(data):
        assert data['age'].dtype == 'int', "Age must be integer!"
        assert data['age'].between(0, 120).all(), "Age out of range!"
        assert data['id'].is_unique, "IDs must be unique!"
        print("All validations passed!")
    

Mini summary: A validation function runs all checks.

Lesson 11: Testing Your Cleaning Pipeline

Definition: Testing your pipeline means running it on data and checking the output.

Why it is important: You need to make sure your pipeline cleans data correctly.

Simple explanation: You run your pipeline on a sample dataset and validate the output.

Real-life example: A data analyst tests a pipeline on a sample dataset.

School example: A student tests a pipeline on a small dataset.

Home example: You test your pipeline on a few days of expenses.

Nigerian example: A business tests a pipeline on a sample sales file.

Illustration:

    cleaned = clean_pipeline(data)
    validate_data(cleaned)
    

Mini summary: Test your pipeline with validation.

Lesson 12: Documenting Data Quality

Definition: Documenting data quality means writing down the validation rules and checks you used.

Why it is important: Documentation helps others trust and understand your data.

Simple explanation: You write a report that explains how you validated the data.

Real-life example: A company documents data quality for compliance.

School example: A student documents data quality for a project.

Home example: You document your budget validation rules.

Nigerian example: A business documents data quality for stakeholders.

Illustration:

    Data Quality Report:
    - Validation rules used
    - Results of checks
    - Any issues found
    - Actions taken
    

Mini summary: Document your validation process.

Lesson 13: Handling Validation Failures

Definition: Handling validation failures means deciding what to do when data fails a check.

Why it is important: You need a plan for when data is not valid.

Simple explanation: If data fails, you can reject it, fix it, or flag it for review.

Real-life example: A company rejects invalid customer records.

School example: A teacher flags invalid grades for review.

Home example: You review invalid expenses.

Nigerian example: A business flags invalid sales for review.

Illustration:

    If validation fails:
    1. Fix the data
    2. Reject the data
    3. Flag for review
    

Mini summary: Have a plan for validation failures.

Lesson 14: Practice Validation

Definition: Practice applying validation techniques to a dataset.

Why it is important: Practice makes you confident in validation.

Simple explanation: You take a dataset and run all validation checks.

Real-life example: A data analyst validates a new dataset.

School example: A student validates a dataset for a project.

Home example: You validate your expense data.

Nigerian example: A student validates local data.

Illustration:

    Step 1: Load data.
    Step 2: Define validation rules.
    Step 3: Run validation.
    Step 4: Review results.
    

Mini summary: Practice validation.

Lesson 15: Your Data Quality Project

Definition: A project where you apply all validation techniques.

Why it is important: It gives you real-world experience.

Simple explanation: You find a dataset, clean it, and validate it.

Real-life example: A data analyst validates a company dataset.

School example: A student completes a class project.

Home example: You validate your own data.

Nigerian example: A student uses a Nigerian dataset.

Illustration:

    Step 1: Load data.
    Step 2: Clean data.
    Step 3: Validate data.
    Step 4: Document results.
    Step 5: Handle issues.
    

Mini summary: Apply validation to a real dataset.

🔑 Key Vocabulary

• Data Quality: How accurate and reliable data is.
• Validation: Checking data against rules.
• Assert: A statement that checks a condition.
• Dimension: An aspect of data quality.
• Accuracy: How correct the data is.
• Completeness: If all data is present.
• Consistency: If data is uniform.
• Timeliness: If data is up-to-date.
• Validity: If data follows rules.
• Documentation: Writing down the process.

💡 Important Concepts

• Data quality has five dimensions.
• Validation checks data against rules.
• Assertions catch errors early.
• Common checks include type, range, uniqueness, and format.
• Write a validation function for all checks.
• Test pipelines with validation.
• Document your validation process.
• Have a plan for validation failures.

📝 Step-by-Step Explanations

How to Validate Data in 6 Steps:

1. Load Data: data = pd.read_csv("file.csv")
2. Define Rules: What should the data look like?
3. Write Asserts: Write assert statements for each rule.
4. Run Validation: Execute your validation code.
5. Handle Errors: Fix any issues found.
6. Document: Write down your validation process.

🌍 Real-life Examples

• A hospital validates patient data to ensure accuracy.
• A school validates student records for completeness.
• A bank validates transaction data for consistency.
• A company validates customer data for validity.

🇳🇬 Nigerian Examples

• A Nigerian bank validates account numbers.
• A Nigerian school validates student IDs.
• A Nigerian business validates product codes.
• A Nigerian hospital validates patient data.

🎈 Fun Examples Children Relate To

• Validating that your list of friends has no duplicates.
• Checking that all ages on a list are reasonable.
• Making sure all phone numbers have 10 digits.
• Checking that all your favorite movies have ratings.

🏠 Everyday Examples

• Checking that your shopping list has no duplicates.
• Validating that your budget adds up correctly.
• Making sure your calendar dates are correct.
• Checking that your contact list has valid email addresses.

🧑‍🏫 Teacher Notes

This module is about quality assurance. Teachers should emphasize the importance of validation in real-world data work. Use examples where poor data quality led to problems. Encourage students to write validation functions for their projects. Highlight the importance of documentation and handling failures.

👪 Parent Tips

Parents can help children understand validation by using everyday examples—like checking that a shopping list has no duplicates or that a budget adds up correctly. Encourage children to think about how they can "validate" their own data at home.

🌟 Interesting Facts

• Data quality is a multi-billion dollar industry.
• Poor data quality costs companies millions every year.
• Data validation is a key part of data governance.
• Validation is critical in healthcare and finance.

🤔 Did You Know?

• Did you know that data quality is one of the most important tasks in data science?
• Did you know that many companies have dedicated data quality teams?
• Did you know that validation is required for regulatory compliance?
• Did you know that a simple assert can save hours of debugging?

🧠 Remember This

• Data quality means data is accurate and reliable.
• Validation checks data against rules.
• Use asserts to catch errors.
• Check data types, ranges, uniqueness, and format.
• Write a validation function for all checks.
• Test pipelines with validation.
• Document your validation process.
• Have a plan for validation failures.

❌ Common Mistakes

• Not validating after cleaning: Assuming data is clean.
• Using wrong data types: Not checking dtypes.
• Forgetting range checks: Allowing out-of-range values.
• Ignoring duplicates: Not checking uniqueness.
• Not documenting: Not writing down validation rules.
• Not handling failures: No plan for invalid data.

✅ Best Practices

• Always validate after cleaning.
• Write validation functions.
• Check data types, ranges, uniqueness, and format.
• Use asserts for validation.
• Test pipelines with validation.
• Document validation rules.
• Have a plan for validation failures.
• Review validation results regularly.

📊 Clear ASCII Illustrations

Validation Process Flowchart

    Data -> Validate -> Valid? -> Yes -> Use Data
                      |
                      No
                      |
                   Fix Data
    

Data Quality Dimensions

    +----------------------------------+
    | Accuracy   | Completeness        |
    | Consistency| Timeliness          |
    | Validity                         |
    +----------------------------------+
    

📋 Comparison Tables

Validation vs Testing

High vs Low Data Quality

📌 End-of-Module Summary

In Module Seven, we learned about data quality, validation, and testing. We discovered that data quality has five dimensions: accuracy, completeness, consistency, timeliness, and validity. We used assertions to validate data against rules like data types, ranges, uniqueness, and format. We created validation functions to organize our checks. We also learned to test pipelines with validation, document the process, and handle validation failures. These skills ensure that our data is clean, reliable, and trustworthy. You are now a master of data quality!

❓ Frequently Asked Questions

1. What is data quality? Data that is accurate and reliable.
2. What are the dimensions of data quality? Accuracy, completeness, consistency, timeliness, validity.
3. What is validation? Checking data against rules.
4. What is an assert? A statement that checks a condition.
5. What are common validation checks? Data type, range, uniqueness, format.
6. What is a validation function? A function that runs all checks.
7. Why test pipelines? To ensure they work correctly.
8. Why document validation? To build trust and understanding.
9. How handle validation failures? Fix, reject, or flag.
10. What is the first step in validation? Define validation rules.

📝 Review Questions

1. What is data quality?
2. What are the five dimensions of data quality?
3. What is validation?
4. What is an assert?
5. What are common validation checks?
6. What is a validation function?
7. Why is testing pipelines important?
8. Why should you document validation?
9. How do you handle validation failures?
10. What is the first step in validation?
11. Give a Nigerian example of validation.
12. Give a home example of validation.
13. What is the difference between validation and testing?
14. What is the benefit of high data quality?
15. What is a common validation check for format?

✍️ Fill-in-the-Blank

1. ______ means data is accurate and reliable. (Data quality)
2. The five dimensions are accuracy, completeness, consistency, timeliness, and ______. (validity)
3. ______ is checking data against rules. (Validation)
4. An ______ statement checks a condition. (assert)
5. Common validation checks include data type, range, uniqueness, and ______. (format)
6. A ______ function runs all validation checks. (validation)
7. You should ______ pipelines with validation. (test)
8. ______ your validation process. (Document)
9. If validation fails, you can fix, reject, or ______ the data. (flag)
10. The first step in validation is to ______ validation rules. (define)

✅ True or False

1. Data quality is not important. (False)
2. Completeness is a dimension of data quality. (True)
3. Validation checks data against rules. (True)
4. An assert statement does nothing. (False)
5. Range check is a common validation. (True)
6. A validation function runs one check. (False)
7. Testing pipelines is not needed. (False)
8. Documentation is optional. (False)
9. You should ignore validation failures. (False)
10. Defining validation rules is the first step. (True)

🔘 Multiple Choice Questions

1. What is data quality?
   A) Data that is fast
   B) Data that is accurate and reliable
   C) Data that is large
   D) Data that is colorful
   Answer: B
2. Which is NOT a dimension of data quality?
   A) Accuracy
   B) Completeness
   C) Speed
   D) Consistency
   Answer: C
3. What is validation?
   A) Deleting data
   B) Checking data against rules
   C) Adding data
   D) Copying data
   Answer: B
4. What does an assert do?
   A) Checks a condition
   B) Deletes data
   C) Adds data
   D) Copies data
   Answer: A
5. Which is a common validation check?
   A) Data type
   B) Range
   C) Uniqueness
   D) All of the above
   Answer: D
6. What is a validation function?
   A) A function that runs all checks
   B) A function that deletes data
   C) A function that adds data
   D) A function that copies data
   Answer: A
7. Why test pipelines?
   A) To make them slower
   B) To ensure they work correctly
   C) To delete data
   D) To copy data
   Answer: B
8. Why document validation?
   A) To confuse people
   B) To build trust
   C) To delete data
   D) To copy data
   Answer: B
9. How handle validation failures?
   A) Ignore them
   B) Fix, reject, or flag
   C) Delete data
   D) Copy data
   Answer: B
10. What is the first step in validation?
    A) Run checks
    B) Define rules
    C) Delete data
    D) Copy data
    Answer: B
11. Which is a Nigerian example of validation?
    A) Validating bank account numbers
    B) Validating movie ratings
    C) Validating video game scores
    D) Validating sports statistics
    Answer: A
12. What is the benefit of high data quality?
    A) Wrong decisions
    B) Good decisions
    C) Confusion
    D) Errors
    Answer: B
13. What is a common format check?
    A) Checking if email has '@'
    B) Checking if age is between 0 and 120
    C) Checking if ID is unique
    D) Checking if data type is correct
    Answer: A
14. What is a common range check?
    A) Checking if email has '@'
    B) Checking if age is between 0 and 120
    C) Checking if ID is unique
    D) Checking if data type is correct
    Answer: B
15. What is a common uniqueness check?
    A) Checking if email has '@'
    B) Checking if age is between 0 and 120
    C) Checking if ID is unique
    D) Checking if data type is correct
    Answer: C

🔗 Matching Exercises

Match the term with its description:

Answers: 1-C, 2-A, 3-B, 4-D, 5-E

✏️ Short Answer Questions

1. What is data quality and why is it important?
2. What are the five dimensions of data quality?
3. What is validation and what does it check?
4. What is a validation function?
5. Give a Nigerian example of validation.

🎭 Scenario-based Exercises

Scenario 1: You have a dataset of students with ages, grades, and emails. Write validation rules for this dataset.

Scenario 2: You clean a dataset and want to validate it. Write a validation function to check completeness and accuracy.

Scenario 3: A validation check fails. What would you do?

👥 Group Activity

In groups, choose a dataset. Write validation rules for it. Create a validation function and test it on the dataset. Discuss any issues and how to handle them.

🧑 Individual Activity

Write a validation function for a dataset of your choice. Include checks for data type, range, uniqueness, and format. Test it on a sample dataset. Write a short report on your results.

💬 Classroom Discussion Questions

• Why is data quality important for businesses?
• What are some challenges in validation?
• How can validation help Nigerian organizations?
• What is the role of documentation in validation?
• How do you handle validation failures?

🛠️ Mini Project

Create a "Data Quality Report" for a dataset of your choice. Include validation rules, results of checks, any issues found, and how they were handled. Show your validation function and results.

📄 Practical Assignment

Find a messy dataset. Clean it using your pipeline. Then, write a validation function and test it on the cleaned data. Write a report on the data quality and any issues found.

🏆 Challenge Exercise

Build a complete data quality framework that includes cleaning, validation, and documentation. Apply it to a dataset and produce a data quality report.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Data quality means data is accurate and reliable.
• The five dimensions are accuracy, completeness, consistency, timeliness, and validity.
• Validation checks data against rules.
• Use asserts to catch errors.
• Common checks: data type, range, uniqueness, format.
• Write a validation function for all checks.
• Test pipelines with validation.
• Document your validation process.
• Have a plan for validation failures.

🔜 Preparation for Module Eight

In Module Eight, we will learn about advanced data cleaning techniques like working with text data, regular expressions, and natural language processing. You will become a true data cleaning expert!

End of Module Seven – You are now a data quality expert!

📘 Module Eight: Python for Data Cleaning – Advanced Text Cleaning with Regular Expressions

Welcome to Module Eight! In previous modules, we learned how to clean numbers, dates, and simple text. But sometimes text is very messy—it has extra spaces, strange characters, or inconsistent patterns. For advanced text cleaning, we use a powerful tool called Regular Expressions (or "regex" for short). Regex helps us find and fix patterns in text. By the end of this module, you will be able to clean even the messiest text data. Let's begin!

🎯 Learning Objectives

After this module, you will be able to:

• Explain what regular expressions are.
• Use common regex patterns to find and clean text.
• Apply regex to extract, replace, and remove text.
• Clean text data using regex in Pandas.
• Apply these techniques to real datasets.

📖 Warm-up Story: The Messy Customer Comments

A company in Nigeria had customer comments. They wanted to find all phone numbers in the comments. But the phone numbers were written in many ways—some with spaces, some with dashes, some with country codes. It was impossible to find them manually.

A data analyst used regular expressions. She wrote a pattern that matched phone numbers in any format—with or without spaces, with or without country code. The regex found all the phone numbers instantly.

This module teaches you how to use regular expressions to clean messy text like a pro.

📚 Main Lessons

Lesson 1: What are Regular Expressions?

Definition: Regular expressions (regex) are patterns used to match and manipulate text.

Why it is important: Regex allows you to find, extract, or replace text based on patterns.

Simple explanation: Think of regex as a super-powered search. It can find text that follows a pattern, like all email addresses or phone numbers.

Real-life example: A company uses regex to extract all email addresses from a customer list.

School example: A student uses regex to find all dates in a document.

Home example: You use regex to find all amounts in your budget.

Nigerian example: A business uses regex to extract phone numbers from customer comments.

Illustration:

    Text: "My phone is 080-123-4567"
    Regex pattern: \d{3}-\d{3}-\d{4}
    Found: 080-123-4567
    

Mini summary: Regex is a powerful tool to find patterns in text.

Lesson 2: Importing the re Module

Definition: The re module is Python's built-in library for regular expressions.

Why it is important: You need to import re to use regex in Python.

Simple explanation: You write: import re

Real-life example: A data analyst imports re before using regex.

School example: A student imports re for a project.

Home example: You import re to clean your data.

Nigerian example: A developer imports re for text processing.

Illustration:

    import re
    

Mini summary: Import re to use regular expressions.

Lesson 3: Basic Regex Patterns

Definition: Basic regex patterns include characters like \d for digits, \w for word characters, and \s for spaces.

Why it is important: These patterns help you match different types of characters.

Simple explanation: \d matches any digit (0-9). \w matches any letter, digit, or underscore. \s matches any space.

Real-life example: You use \d to find all numbers in a text.

School example: You use \w to find all words.

Home example: You use \s to find spaces.

Nigerian example: You use \d to find phone digits.

Illustration:

    \d -> matches any digit
    \w -> matches any word character
    \s -> matches any whitespace
    

Mini summary: Basic patterns match digits, words, and spaces.

Lesson 4: Quantifiers – How Many?

Definition: Quantifiers specify how many times a pattern should appear—like +, *, and ?.

Why it is important: They help you match patterns of varying length.

Simple explanation: + means "one or more", * means "zero or more", ? means "zero or one".

Real-life example: You use \d+ to match one or more digits.

School example: You use \s* to match zero or more spaces.

Home example: You use \w? to match zero or one word character.

Nigerian example: You use \d{3} to match exactly 3 digits.

Illustration:

    + -> one or more
    * -> zero or more
    ? -> zero or one
    {3} -> exactly 3
    {2,4} -> 2 to 4
    

Mini summary: Quantifiers control how many times a pattern appears.

Lesson 5: Special Characters and Escaping

Definition: Special characters like ., *, +, ?, etc., have special meanings in regex. To match them literally, you need to escape them with a backslash (\).

Why it is important: If you want to match a period (.) as a period, you must escape it.

Simple explanation: To match a literal dot, you use \.

Real-life example: You use \. to match a period in a file name.

School example: You use \* to match an asterisk.

Home example: You use \? to match a question mark.

Nigerian example: You use \. to match a decimal point.

Illustration:

    . -> any character (except newline)
    \. -> a literal period
    * -> zero or more
    \* -> a literal asterisk
    

Mini summary: Escape special characters to match them literally.

Lesson 6: Searching with re.search()

Definition: re.search() searches for a pattern in a string and returns the first match.

Why it is important: It's useful when you just need to know if a pattern exists.

Simple explanation: re.search(pattern, text) returns a match object if found, else None.

Real-life example: You check if a text contains an email address.

School example: You check if a document contains a date.

Home example: You check if a note contains a phone number.

Nigerian example: You check if a comment contains a phone number.

Illustration:

    result = re.search(r'\d{3}-\d{3}-\d{4}', text)
    if result:
        print("Found:", result.group())
    

Mini summary: re.search() finds the first match of a pattern.

Lesson 7: Finding All Matches with re.findall()

Definition: re.findall() returns all non-overlapping matches of a pattern in a string.

Why it is important: It's useful when you want to extract all occurrences of a pattern.

Simple explanation: re.findall(pattern, text) returns a list of all matches.

Real-life example: You extract all email addresses from a text.

School example: You extract all dates from a document.

Home example: You extract all numbers from a budget.

Nigerian example: You extract all phone numbers from customer comments.

Illustration:

    emails = re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', text)
    

Mini summary: re.findall() returns all matches.

Lesson 8: Replacing Text with re.sub()

Definition: re.sub() replaces occurrences of a pattern with a replacement string.

Why it is important: It's useful for cleaning text by removing or replacing unwanted patterns.

Simple explanation: re.sub(pattern, replacement, text) returns the text with replacements.

Real-life example: You remove extra spaces from text.

School example: You replace abbreviations with full words.

Home example: You remove special characters from a string.

Nigerian example: You replace inconsistent date formats.

Illustration:

    clean_text = re.sub(r'\s+', ' ', text)  # Replace multiple spaces with one
    

Mini summary: re.sub() replaces patterns in text.

Lesson 9: Splitting Text with re.split()

Definition: re.split() splits a string by the occurrences of a pattern.

Why it is important: It's useful for breaking text into parts based on a delimiter.

Simple explanation: re.split(pattern, text) returns a list of split parts.

Real-life example: You split a sentence by commas.

School example: You split a CSV line by commas.

Home example: You split a grocery list by commas.

Nigerian example: You split customer data by semicolons.

Illustration:

    parts = re.split(r',\s*', text)
    

Mini summary: re.split() splits text by a pattern.

Lesson 10: Using Regex in Pandas

Definition: You can use regex in Pandas with methods like str.contains(), str.extract(), and str.replace().

Why it is important: It allows you to clean text data in DataFrames efficiently.

Simple explanation: You use data['col'].str.contains(pattern) to find rows matching a pattern.

Real-life example: You extract all phone numbers from a customer column.

School example: You extract all student IDs.

Home example: You extract all amounts from a budget column.

Nigerian example: You extract all email addresses from a customer list.

Illustration:

    data['phone'] = data['comment'].str.extract(r'(\d{3}-\d{3}-\d{4})')
    

Mini summary: Use regex in Pandas with str methods.

Lesson 11: Common Text Cleaning Tasks with Regex

Definition: Common tasks include removing extra spaces, extracting phone numbers, cleaning emails, and removing special characters.

Why it is important: These are the most frequent cleaning tasks in real-world data.

Simple explanation: You use regex to find and fix these common problems.

Real-life example: A company removes extra spaces from customer names.

School example: A student extracts dates from a text.

Home example: You remove special characters from a list.

Nigerian example: A business extracts phone numbers from comments.

Illustration:

    # Remove extra spaces
    data['text'] = data['text'].str.replace(r'\s+', ' ', regex=True)
    # Extract phone numbers
    data['phone'] = data['text'].str.extract(r'(\d{3}-\d{3}-\d{4})')
    

Mini summary: Regex handles common text cleaning tasks.

Lesson 12: Writing Complex Regex Patterns

Definition: Complex patterns combine multiple elements to match intricate text patterns.

Why it is important: Some patterns are not simple—like matching emails or URLs.

Simple explanation: You combine characters, quantifiers, and groups to build a pattern.

Real-life example: A pattern to match Nigerian phone numbers.

School example: A pattern to match student IDs.

Home example: A pattern to match dates.

Nigerian example: A pattern to match Lagos addresses.

Illustration:

    # Nigerian phone number pattern
    pattern = r'(0|\+234)[7-9][0-9]{9}'
    

Mini summary: Complex patterns combine many elements.

Lesson 13: Practice Cleaning Text with Regex

Definition: Practice applying regex to clean messy text.

Why it is important: Practice helps you master regex.

Simple explanation: You take a messy text column and apply regex to clean it.

Real-life example: A data analyst cleans customer comments.

School example: A student cleans a text dataset.

Home example: You clean your own text data.

Nigerian example: A student cleans local text data.

Illustration:

    data['clean_text'] = data['text'].str.replace(r'[^a-zA-Z0-9\s]', '', regex=True)
    

Mini summary: Practice regex on real text data.

Lesson 14: Validating Text with Regex

Definition: You can use regex to validate text patterns, like checking if a string is a valid email or phone number.

Why it is important: It ensures text data meets expected formats.

Simple explanation: You check if a string matches a pattern.

Real-life example: A company validates email addresses.

School example: A student validates student IDs.

Home example: You validate phone numbers.

Nigerian example: A business validates NIN numbers.

Illustration:

    def is_valid_email(email):
        pattern = r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'
        return re.match(pattern, email) is not None
    

Mini summary: Regex validates text patterns.

Lesson 15: Your Text Cleaning Project

Definition: A project where you apply regex to clean a text dataset.

Why it is important: It gives you real-world experience.

Simple explanation: You find a messy text dataset and clean it using regex.

Real-life example: A data analyst cleans customer comments.

School example: A student completes a class project.

Home example: You clean your own text data.

Nigerian example: A student uses a Nigerian text dataset.

Illustration:

    Step 1: Load text data.
    Step 2: Identify patterns.
    Step 3: Apply regex to clean.
    Step 4: Validate results.
    

Mini summary: Apply regex to a real text cleaning project.

🔑 Key Vocabulary

• Regular Expression (regex): A pattern to match text.
• re: Python's regular expression module.
• Pattern: The regex string used to match text.
• Quantifier: Specifies how many times a pattern appears.
• Escape: Using a backslash to match special characters.
• search(): Finds the first match.
• findall(): Finds all matches.
• sub(): Replaces matches.
• split(): Splits text by pattern.
• Valid: Data that follows a pattern.

💡 Important Concepts

• Regex is for pattern matching in text.
• Import re to use regex.
• Use quantifiers for repetition.
• Escape special characters to match them.
• Use search(), findall(), sub(), split().
• Use regex in Pandas with str methods.
• Regex is used for cleaning, extracting, and validating.

📝 Step-by-Step Explanations

How to Clean Text with Regex in 6 Steps:

1. Import re: import re
2. Define Pattern: pattern = r'...'
3. Search or Extract: re.search(pattern, text)
4. Replace: re.sub(pattern, replacement, text)
5. Apply to DataFrame: data['col'].str.replace(pattern, ...)
6. Validate: Check if text matches pattern.

🌍 Real-life Examples

• A company extracts email addresses from text.
• A school cleans student comments.
• A hospital extracts patient IDs.
• A bank validates account numbers.

🇳🇬 Nigerian Examples

• A Nigerian business extracts phone numbers from customer comments.
• A Nigerian school validates student IDs.
• A Nigerian hospital validates NIN numbers.
• A Nigerian government cleans address data.

🎈 Fun Examples Children Relate To

• Extract all video game titles from a text.
• Clean a list of your favorite movies.
• Extract all dates from your calendar.
• Clean your to-do list by removing special characters.

🏠 Everyday Examples

• Remove extra spaces from your notes.
• Extract phone numbers from your contacts.
• Clean a grocery list.
• Validate email addresses.

🧑‍🏫 Teacher Notes

This module introduces regular expressions, a powerful but sometimes tricky topic. Teachers should start with simple patterns and gradually introduce complexity. Use plenty of examples and practice exercises. Emphasize that regex is a tool for pattern matching, and it's okay to start with simple patterns.

👪 Parent Tips

Parents can help children understand regex by using everyday examples—like searching for a specific word in a book or finding numbers in a text. Encourage children to think about patterns in text and how they can be used to clean data.

🌟 Interesting Facts

• Regular expressions were invented in the 1950s.
• Regex is used in almost every programming language.
• Regex can be very complex and powerful.
• Regex is often used to validate user input.

🤔 Did You Know?

• Did you know that regex is used in search engines?
• Did you know that regex can be used to parse HTML?
• Did you know that regex is a key skill for data scientists?
• Did you know that regex can be slow for large text?

🧠 Remember This

• Regex is for pattern matching.
• Import re to use regex.
• Use quantifiers for repetition.
• Escape special characters.
• Use search(), findall(), sub(), split().
• Apply regex in Pandas with str methods.
• Regex can extract, replace, and validate text.

❌ Common Mistakes

• Not escaping special characters: Forgetting to escape . or *.
• Using wrong quantifiers: Using + instead of *.
• Not using raw strings: Forgetting r before pattern.
• Overcomplicating: Making patterns too complex.
• Not testing: Not testing regex on sample text.

✅ Best Practices

• Use raw strings for patterns: r'...'.
• Start with simple patterns and build complexity.
• Test regex on sample text before applying to full data.
• Use comments to document complex patterns.
• Use regex in Pandas for efficient text cleaning.
• Validate results after cleaning.

📊 Clear ASCII Illustrations

Regex Matching Process

    Text -> Pattern -> Match? -> Yes -> Extract/Replace
                          |
                          No
                          |
                   No Action
    

Common Regex Patterns

    \d  -> digit
    \w  -> word character
    \s  -> whitespace
    .   -> any character
    +   -> one or more
    *   -> zero or more
    ?   -> zero or one
    {3} -> exactly 3
    

📋 Comparison Tables

Regex Methods

Without Regex vs With Regex

📌 End-of-Module Summary

In Module Eight, we learned about regular expressions (regex) for advanced text cleaning. We discovered what regex is and how to use it to find patterns in text. We learned basic patterns, quantifiers, and how to escape special characters. We used re.search(), re.findall(), re.sub(), and re.split() to manipulate text. We also applied regex in Pandas using str methods. We practiced common text cleaning tasks like removing extra spaces, extracting phone numbers, and validating email addresses. These skills are essential for cleaning messy text data. You are now a text-cleaning expert!

❓ Frequently Asked Questions

1. What is regex? A pattern to match text.
2. How do you import regex? import re
3. What is \d? Matches a digit.
4. What is +? One or more.
5. What is *? Zero or more.
6. What is ? Zero or one.
7. What does re.search() do? Finds first match.
8. What does re.findall() do? Finds all matches.
9. What does re.sub() do? Replaces matches.
10. Can regex be used in Pandas? Yes, with str methods.

📝 Review Questions

1. What is a regular expression?
2. How do you import the regex module?
3. What is the pattern for a digit?
4. What does + mean in regex?
5. What does * mean in regex?
6. What does ? mean in regex?
7. What does re.search() do?
8. What does re.findall() do?
9. What does re.sub() do?
10. What does re.split() do?
11. How do you use regex in Pandas?
12. Give a Nigerian example of using regex.
13. Give a home example of using regex.
14. What is the benefit of using regex?
15. What is a common text cleaning task with regex?

✍️ Fill-in-the-Blank

1. ______ are patterns to match text. (Regular expressions)
2. Use ______ to import the regex module. (import re)
3. ______ matches any digit. (\d)
4. ______ means one or more. (+)
5. ______ means zero or more. (*)
6. ______ means zero or one. (?)
7. ______ finds the first match. (re.search())
8. ______ finds all matches. (re.findall())
9. ______ replaces matches. (re.sub())
10. ______ splits text by pattern. (re.split())

✅ True or False

1. Regex is only for numbers. (False)
2. re is Python's regex module. (True)
3. \d matches a letter. (False)
4. + means zero or more. (False)
5. * means zero or more. (True)
6. ? means zero or one. (True)
7. re.search() finds all matches. (False)
8. re.findall() finds all matches. (True)
9. re.sub() replaces matches. (True)
10. Regex cannot be used in Pandas. (False)

🔘 Multiple Choice Questions

1. What is regex?
   A) A type of data
   B) A pattern to match text
   C) A Python library
   D) A file format
   Answer: B
2. How do you import regex?
   A) import re
   B) import regex
   C) import rex
   D) import regular
   Answer: A
3. What is \d?
   A) A digit
   B) A letter
   C) A space
   D) A special character
   Answer: A
4. What does + mean?
   A) Zero or more
   B) One or more
   C) Zero or one
   D) Exactly one
   Answer: B
5. What does * mean?
   A) Zero or more
   B) One or more
   C) Zero or one
   D) Exactly one
   Answer: A
6. What does ? mean?
   A) Zero or more
   B) One or more
   C) Zero or one
   D) Exactly one
   Answer: C
7. What does re.search() do?
   A) Finds first match
   B) Finds all matches
   C) Replaces matches
   D) Splits text
   Answer: A
8. What does re.findall() do?
   A) Finds first match
   B) Finds all matches
   C) Replaces matches
   D) Splits text
   Answer: B
9. What does re.sub() do?
   A) Finds first match
   B) Finds all matches
   C) Replaces matches
   D) Splits text
   Answer: C
10. What does re.split() do?
    A) Finds first match
    B) Finds all matches
    C) Replaces matches
    D) Splits text
    Answer: D
11. Which method is used in Pandas for regex?
    A) str.contains()
    B) str.extract()
    C) str.replace()
    D) All of the above
    Answer: D
12. Which is a Nigerian example of regex?
    A) Extracting phone numbers
    B) Extracting movie titles
    C) Extracting video game scores
    D) Extracting sports statistics
    Answer: A
13. What is a common regex task?
    A) Removing extra spaces
    B) Adding numbers
    C) Creating graphs
    D) Analyzing data
    Answer: A
14. What is the benefit of regex?
    A) Fast pattern matching
    B) Slow execution
    C) Complex syntax
    D) Limited use
    Answer: A
15. What is the first step in regex?
    A) Define pattern
    B) Import re
    C) Apply to data
    D) Validate results
    Answer: B

🔗 Matching Exercises

Match the term with its description:

Answers: 1-C, 2-A, 3-B, 4-D, 5-E

✏️ Short Answer Questions

1. What is regex and why is it useful?
2. What is the difference between re.search() and re.findall()?
3. How do you replace text using regex?
4. Give a Nigerian example of using regex.
5. What is the benefit of using regex in Pandas?

🎭 Scenario-based Exercises

Scenario 1: You have a dataset of customer comments. Extract all phone numbers using regex.

Scenario 2: You have a dataset of addresses. Clean the addresses by removing extra spaces and special characters.

Scenario 3: You have a dataset of emails. Validate that all emails are in the correct format.

👥 Group Activity

In groups, find a messy text dataset. Identify common patterns (phone numbers, emails, dates). Write regex patterns to extract and clean these patterns. Present your results to the class.

🧑 Individual Activity

Write a Python script that uses regex to clean a text column in a dataset. Include extraction, replacement, and validation. Test it on a sample dataset. Write a short report on your results.

💬 Classroom Discussion Questions

• Why is regex important for data cleaning?
• What are some challenges in writing regex patterns?
• How can regex help Nigerian businesses?
• What are the limitations of regex?
• How can you practice and improve your regex skills?

🛠️ Mini Project

Create a "Text Cleaning Tool" that uses regex to clean a text column. Include functions to remove extra spaces, extract phone numbers, and validate emails. Apply it to a real dataset and show the results.

📄 Practical Assignment

Find a messy text dataset (e.g., customer comments, social media posts). Clean it using regex. Write a report on your cleaning process, including the patterns you used and the results.

🏆 Challenge Exercise

Write a regex pattern to match Nigerian phone numbers in all common formats (with spaces, dashes, country code). Test it on a large text dataset and extract all phone numbers.

🔍 Quiz Answers

Answers to the Multiple Choice Questions are already provided above. For other exercises, refer to the lessons and key vocabulary.

🎁 Key Takeaways

• Regex is a powerful tool for pattern matching in text.
• Import re to use regex.
• Use quantifiers like +, *, ?.
• Escape special characters with \.
• Use search(), findall(), sub(), split().
• Apply regex in Pandas with str methods.
• Regex can extract, replace, and validate text.

🔜 Preparation for Module Nine

In Module Nine, we will learn about data integration and merging from multiple sources. You will learn how to combine data from different files and databases. Get ready for the final module!

End of Module Eight – You are now a regex master!