Python Pandas - Dropping Missing Data

---

---

Missing data is a common issue when working with real-world datasets. The Python Pandas library provides an easy way for removing rows or columns that contain missing values (NaN or NaT) from a dataset using the dropna() method.

The dropna() method in Pandas is a useful tool to handle missing data by dropping rows or columns based on your specific requirements. In this tutorial, we will learn how to use dropna() to clean your dataset by dropping missing data based on various conditions.

The dropna() Method

The Pandas dropna() method allows you to remove missing values from a Pandas data structures such as, Series and DataFrame objects. It offers several options to customize how you drop rows or columns based on the presence of NaN values. This method returns a new Pandas object with missing data dropped or it returns None if inplace parameter is set to True.

Syntax

Following is the syntax −

 DataFrame.dropna(*, axis=0, how=<no_default>, thresh=<no_default>, subset=None, inplace=False, ignore_index=False) 

Where,

• axis: 0 or 'index' (default) to drop rows; 1 or 'columns' to drop columns.

• how: By default it is set to 'any', which drops that row or column if any missing values are present. If set to 'all', then it drops that row or column if all the missing values.

• thresh: Require a minimum number of non-NA values to retain the row or column.

• subset: List of specific columns (if dropping rows) or rows (if dropping columns) to consider.

• inplace: Modify the DataFrame in place (default is False).

• ignore_indexReset the index of the result (default is False).

Let's explore the how of the dropna() method drops the missing data based on various conditions.

Drop Rows with Any Missing Values

By default, the dropna() method removes rows where any missing values are present.

Example

The following example uses the dropna() method to drop the rows that have any missing values.

 import pandas as pd import numpy as np dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll_number": [23, 45, np.nan, 18], "Major_Subject": ["Maths", "Physics", "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]} df = pd.DataFrame(dataset, index= [1, 2, 3, 4]) print("Original DataFrame:") print(df) # Drop the rows that have any missing values df_cleaned = df.dropna() print('\nResultant DataFrame after removing row:\n',df_cleaned) 

Following is the output of the above code −

 Original DataFrame: 

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	Krishna	45.0	Physics	NaN
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Resultant DataFrame after removing row:

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0

Drop Rows Where All Values Are Missing

To drop rows where all values are missing, then we need to set how='all' parameter to the dropna() method.

Example

The following example demonstrates how to drop the rows where all values are missing in a DataFrame.

 import pandas as pd import numpy as np dataset = {"Student name": ["Ajay", np.nan, "Deepak", "Swati"], "Roll number": [23, np.nan, np.nan, 18], "Major Subject": ["Maths", np.nan, "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]} df = pd.DataFrame(dataset, index= [1, 2, 3, 4]) print("Original DataFrame:") print(df) # Drop rows where all values are missing reslut = df.dropna(how='all') print('\nResultant DataFrame after removing row:\n',reslut) 

Following is the output of the above code −

 Original DataFrame: 

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	NaN	NaN	NaN	NaN
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Resultant DataFrame after removing row:

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Keep Rows with a Minimum Number of Missing Values

The pandas dropan() method provides the thresh parameter to specify a minimum threshold of non-missing values for keeping rows with a minimum number of Non-Na values.

Example

This example demonstrates how to keep the rows the minimum number of missing values.

 import pandas as pd import numpy as np dataset = {"Student name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll number": [23, np.nan, np.nan, 18], "Major Subject": ["Maths", np.nan, "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]} df = pd.DataFrame(dataset, index= [1, 2, 3, 4]) print("Original DataFrame:") print(df) # Drop the rows with a threshold result = df.dropna(thresh=2) print('\nResultant DataFrame after removing row:\n',result) 

Following is the output of the above code −

 Original DataFrame: 

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	Krishna	NaN	NaN	NaN
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Resultant DataFrame after removing row:

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Drop Columns with Any Missing Values

To drop columns that contain any missing values, then we can use axis parameter of the dropna() method to select the columns.

Example

This example show how the dropna() method removes the entire column where any of the value is missing.

 import pandas as pd import numpy as np dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll_number": [23, 45, np.nan, 18], "Major_Subject": ["Maths", "Physics", "Arts", "Political science"], "Marks": [57, np.nan, 98, np.nan]} df = pd.DataFrame(dataset, index= [1, 2, 3, 4]) print("Original DataFrame:") print(df) # Drop column with any missing values result = df.dropna(axis='columns') print('\nResultant DataFrame after removing columns:\n',result) 

Following is the output of the above code −

 Original DataFrame: 

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	Krishna	45.0	Physics	NaN
3	Deepak	NaN	Arts	98.0
4	Swati	18.0	Political science	NaN

Resultant DataFrame after removing columns:

	Student name	Major Subject
1	Ajay	Maths
2	Krishna	Physics
3	Deepak	Arts
4	Swati	Political science

Drop Rows Based on Missing Data in Specific Columns

You can use the subset parameter of the drop() method to focus only on those particular columns while dropping rows where data is missing.

Example

This example shows how to remove the rows based on missing data present in the specific column using the subset parameter of the dropna() method.

 import pandas as pd import numpy as np dataset = {"Student_name": ["Ajay", "Krishna", "Deepak", "Swati"], "Roll_number": [23, 45, np.nan, 18], "Major_Subject": ["Maths", "Physics", np.nan, "Political science"], "Marks": [57, np.nan, 98, np.nan]} df = pd.DataFrame(dataset, index= [1, 2, 3, 4]) print("Original DataFrame:") print(df) # Drop Rows Based on Missing Data in Specific Columns result = df.dropna(subset=['Roll_number', 'Major_Subject']) print('\nResultant DataFrame after removing rows:\n',result) 

Following is the output of the above code −

 Original DataFrame: 

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	Krishna	45.0	Physics	NaN
3	Deepak	NaN	NaN	98.0
4	Swati	18.0	Political science	NaN

Resultant DataFrame after removing rows:

	Student name	Roll number	Major Subject	Marks
1	Ajay	23.0	Maths	57.0
2	Krishna	45.0	Physics	NaN
4	Swati	18.0	Political science	NaN