Python Pandas - Iteration

Iterating over pandas objects is a fundamental task in data manipulation, and the behavior of iteration depends on the type of object you're dealing with. This tutorial explains how iteration works in pandas, specifically focusing on Series and DataFrame objects.

The iteration behavior in pandas varies between Series and DataFrame objects −

• Series: Iterating over a Series object yields the values directly, making it similar to an array-like structure.

• DataFrame: Iterating over a DataFrame follows a dictionary-like convention, where the iteration produces the column labels (i.e., the keys).

Iterating Through Rows in a DataFrame

To iterate over the rows of the DataFrame, we can use the following methods −

• items(): to iterate over the (key,value) pairs

• iterrows(): iterate over the rows as (index,series) pairs

• itertuples(): iterate over the rows as namedtuples

Iterate Over Column Pairs

The items() method allows you to iterate over each column as a key-value pair, with the label as the key and the column values as a Series object. This method is consistent with the dictionary-like interface of a DataFrame.

Example

The following example iterates a DataFrame rows using the items() method. In this example each column is iterated separately as a key-value pair in a Series.

 import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(4,3),columns=['col1','col2','col3']) print("Original DataFrame:\n", df) # Iterate Through DataFrame rows print("Iterated Output:") for key,value in df.items(): print(key,value) 

Its output is as follows −

 Original DataFrame: col1 col2 col3 0 0.422561 0.094621 -0.214307 1 0.430612 -0.334812 -0.010867 2 0.350962 -0.145470 0.988463 3 1.466426 -1.258297 -0.824569 Iterated Output: col1 0 0.422561 1 0.430612 2 0.350962 3 1.466426 Name: col1, dtype: float64 col2 0 0.094621 1 -0.334812 2 -0.145470 3 -1.258297 Name: col2, dtype: float64 col3 0 -0.214307 1 -0.010867 2 0.988463 3 -0.824569 Name: col3, dtype: float64

Observe, each column is iterated separately, where key is the column name, and value is the corresponding Series object.

Iterate Over DataFrame as Series Pairs

The iterrows() method returns an iterator that yields index and row pairs, where each row is represented as a Series object, containing the data in each row.

Example

The following example iterates the DataFrame rows using the iterrows() method.

 import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(4,3),columns = ['col1','col2','col3']) print("Original DataFrame:\n", df) # Iterate Through DataFrame rows print("Iterated Output:") for row_index,row in df.iterrows(): print(row_index,row) 

Its output is as follows −

 Original DataFrame: col1 col2 col3 0 0.468160 -0.634193 -0.603612 1 1.231840 0.090565 -0.449989 2 -1.645371 0.032578 -0.165950 3 1.956370 -0.261995 2.168167 Iterated Output: 0 col1 0.468160 col2 -0.634193 col3 -0.603612 Name: 0, dtype: float64 1 col1 1.231840 col2 0.090565 col3 -0.449989 Name: 1, dtype: float64 2 col1 -1.645371 col2 0.032578 col3 -0.165950 Name: 2, dtype: float64 3 col1 1.956370 col2 -0.261995 col3 2.168167 Name: 3, dtype: float64

Note: Because iterrows() iterate over the rows, it doesn't preserve the data type across the row. 0,1,2 are the row indices and col1,col2,col3 are column indices.

Iterate Over DataFrame as Namedtuples

The itertuples() method will return an iterator yielding a named tuple for each row in the DataFrame. The first element of the tuple will be the rows corresponding index value, while the remaining values are the row values. This method is generally faster than iterrows() and preserves the data types of the row elements.

Example

The following example uses the itertuples() method to loop thought a DataFrame rows as Namedtuples

 import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(4,3),columns = ['col1','col2','col3']) print("Original DataFrame:\n", df) # Iterate Through DataFrame rows print("Iterated Output:") for row in df.itertuples(): print(row) 

Its output is as follows −

 Original DataFrame: col1 col2 col3 0 0.501238 -0.353269 -0.058190 1 -0.426044 -0.012733 -0.532594 2 -0.704042 2.201186 -1.960429 3 0.514151 -0.844160 0.508056 Iterated Output: Pandas(Index=0, col1=0.5012381423628608, col2=-0.3532690739340918, col3=-0.058189913290578134) Pandas(Index=1, col1=-0.42604395958954777, col2=-0.012733326002509393, col3=-0.5325942971498149) Pandas(Index=2, col1=-0.7040424042099052, col2=2.201186165472291, col3=-1.9604285032438307) Pandas(Index=3, col1=0.5141508750506754, col2=-0.8441600001815068, col3=0.5080555294913854) 

Iterating Through DataFrame Columns

When you iterate over a DataFrame, it will simply returns the column names.

Example

Let us consider the following example to understand the iterate over a DataFrame columns.

 import pandas as pd import numpy as np N = 5 df = pd.DataFrame({ 'A': pd.date_range(start='2016-01-01', periods=N, freq='D'), 'x': np.linspace(0, stop=N-1, num=N), 'y': np.random.rand(N), 'C': np.random.choice(['Low', 'Medium', 'High'], N).tolist(), 'D': np.random.normal(100, 10, size=N).tolist() }) print("Original DataFrame:\n", df) # Iterate Through DataFrame Columns print("Output:") for col in df: print(col) 

Its output is as follows −

 Original DataFrame: A x y C D 0 2016-01-01 0.0 0.990949 Low 114.143838 1 2016-01-02 1.0 0.314517 High 95.559640 2 2016-01-03 2.0 0.180237 Low 121.134817 3 2016-01-04 3.0 0.170095 Low 95.643132 4 2016-01-05 4.0 0.920718 Low 96.379692 Output: A x y C D 

Example

While iterating over a DataFrame, you should not modify any object. Iteration is meant for reading, and the iterator returns a copy of the original object (a view), meaning changes will not reflect on the original object. The following example demonstrates the above statement.

 import pandas as pd import numpy as np df = pd.DataFrame(np.random.randn(4,3),columns = ['col1','col2','col3']) for index, row in df.iterrows(): row['a'] = 10 print(df) 

Its output is as follows −

 col1 col2 col3 0 -1.739815 0.735595 -0.295589 1 0.635485 0.106803 1.527922 2 -0.939064 0.547095 0.038585 3 -1.016509 -0.116580 -0.523158 

As you can see, no changes are reflected in the DataFrame since the iteration only provides a view of the data.