Programming Level-up
An Introduction to Pandas
Table of Contents
1. Pandas
1.1. Introduction
1.1.1. What is Pandas?
- Pandas a library to make the representation and manipulation of tabular data easier in Python.
- A table of data is called a 'Dataframe' that consists of named columns and (optionally) named rows.
- https://pandas.pydata.org/
1.1.2. Installing and importing pandas
To install pandas, we can either use conda:
conda install pandas
or with pip:
pip install pandas
After pandas has been installed. We shall import it into our scripts (using the common convention of aliasing the library as pd):
import pandas as pd
1.1.3. Creating a dataframe
Now that pandas has been successfully imported, we're ready to create and manipulate our own dataframes. To create a dataframe, we first need to organise our data in appropriate format. Perhaps one of the most simple formats for this data is a dictionary, where each value is a list:
data = {"col1": [1, 2], "col2": [3, 4]}
We see that each 'key' is the representation of a column of data, and the value of this key is a list of data for this column. To convert this data to a dataframe, we need only to call the DataFrame class:
df = pd.DataFrame(data)
1.1.4. Creating a dataframe
df
(dataframe for short) is now our representation of the dataframe:
We see that each column is named using the keys in our data
dictionary, and the
values of the column correspond to the elements in the list. To the left of the
dataframe we have a numerical index starting at 0.
1.1.5. Access elements in our dataframe
Extracting particular values from this dataframe can be accomplished using the
loc
and iloc
class methods. First let's look at using loc
, and later on we'll
investigate the differences between these two methods.
Let's say we want to get all the data for the first row of our dataframe:
df.loc[0]
This returns a 'Series', which is just a representation of a vector of data.
1.1.6. Access elements in our dataframe
To access a single value from this series, we can specify the column name:
df.loc[0]["col1"] # returns one
Or, we can more simply add the column name into the loc
:
df.loc[0, "col1"]
If we wanted to retrieve a subset of columns, we supply a list of column names:
df.loc[0, ["col1", "col2"]]
1.1.7. Access elements in our dataframe
We can also use the slice notation to access multiple rows:
df.loc[0:2, "col1"]
This retrieves the values in col1
.
Or if we just wanted to get the entire column of data, we could instead do:
df["col1"]
1.2. Manipulating data
1.2.1. Reading a CSV file
Instead of manually constructing our data and then passing it to a DataFrame, we can use pandas to read directly from a CSV file and return a DataFrame:
Let's say we have a CSV file of measurements of Iris flowers called iris.csv
. We
can read this CSV file using the pd.read_csv
method.
df = pd.read_csv("iris.csv")
1.2.2. Selecting a subset of data
With this more complex dataset, we can use more fancy methods of indexing. For example, let's select all the rows where the sepal length is less than 5 cm.
df[df["sepal length (cm)"] < 5]
Instead of the 150 rows we had before, this returns just 22. We can also specify only the columns we want with this conditional expression:
df[df["sepal length (cm)"] < 5]["sepal width (cm)"]
1.2.3. Creating new columns
We can add new columns to this dataset by using the assignment operator. In this example, we're creating a new column called 'sepal sum' to be the sum of both the 'sepal width' and 'sepal length':
df["sepal sum"] = df["sepal width (cm)"] + df["sepal length (cm)"]
1.3. Inspecting our data
1.3.1. Shape of the data
We can also further see that our new column has been added by inspecting the
shape
of the data.
df.shape
(150, 5)
This returns a tuple corresponding to the number of rows (150) and the number of columns (5).
1.3.2. Getting the names of columns
To find out what the names of the columns are we can use the columns
attribute:
df.columns
Index(['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)', 'sepal sum'], dtype='object')
This returns an Index that can itself be indexed in the usual way:
df.columns[0]
'sepal length (cm)'
1.3.3. Head/tail
We can get the first/last few rows of the data using the .head()
or .tail()
methods. These take an optional argument specifying the number of rows to
view. By default, it will show 10 rows.
df.head() # shows the first 10 rows df.head(5) # shows the first 5 rows df.tail() # shows the last 10 rows df.tail(5) # shows the last 5 rows
1.4. Operations
1.4.1. Operations on data
Pandas comes with a few standard methods to perform some basic operations. For example, you can calculate the mean of a column:
df["sepal length (cm)"].mean()
And you can use the apply()
method to apply a function to every element
(i.e. map a function to every element):
df["sepal length (cm)"].apply(lambda x: x * 2)
Apply takes a function as an argument, and here we're using an anonymous (unnamed function) using a lambda expression https://docs.python.org/3/tutorial/controlflow.html#lambda-expressions
This lambda expression will double its input, and therefore applying this function to every element will double all values in 'sepal length (cm)'.
1.4.2. Merge
Many pandas dataframes can be combined together using the concat
method that
requires a list of dataframes as input.
data1 = pd.DataFrame({"col1": [0, 1], "col2": [0, 1]}) data2 = pd.DataFrame({"col1": [2, 3], "col2": [2, 3]}) combined = pd.concat([data1, data2])
1.4.3. More on indexing
Notice how the indexes are repeated. We can also verify this using the .index
attribute:
combined.index
Int64Index([0, 1, 0, 1], dtype='int64')
We can see two '0's and two '1's. Normally, this is not a problem, but it does
have an effect on when we index our data with loc
.
1.4.4. More on indexing
combined.loc[1]
Notice how loc
has returned two rows because it sees two rows with the index
label of 1. If instead we simply meant: give me the second row we should use
iloc
:
combined.iloc[1]
Which will give us the desired outcome.
1.4.5. Resetting indexes
Alternatively we can reset the index labels:
combined.reset_index()
This will compute a new series of indexes for our data, and then using loc
again
will only return the one row.
1.4.6. Resetting indexes
To save the result of reset_index()
we need to overwrite our original data:
combined = combined.reset_index()
Or specify inplace
:
combined.reset_index(inplace=True)
1.5. Different types of data
1.5.1. Categorical data
So far, we've only seen numerical data. One of the advantages of using pandas for tabular data is that we can represent various other types of data that makes our manipulation and operations on different data types simpler. For example, we can represent 'categorical data' where there is a finite set of values or categories.
df = pd.DataFrame({"col1": ["a", "b", "c", "a"], "col2": [1, 2, 5, 4]})
Right now, df
is simply representing 'col1' as strings, but we can change the
representation to categorical elements with:
df["col1"] = df["col1"].astype("category")
1.5.2. Categorical data
With categorical data, we can perform operations on these groups a lot quicker than if we were just to represent them on strings. For instance, lets compute the sum of 'col2' for each group.
df.groupby("col1").sum()
If we have lots of data, having 'col1' astype('category')
will be a lot more
computationally efficient than leaving them as strings.
1.5.3. Dates and times
If you have a column that represents a date or time, you can convert that column
to a true datetime representation with pd.to_datetime
df = pd.DataFrame({"col1": ["2002/01/30", "2010/05/16"]}) df["col1"] = pd.to_datetime(df["col1"])
In addition to make indexing by dates a lot faster, it also provides us with some convienant methods to extract particular components from the data. Such as the year:
df["col1"].dt.year # or df["col1"].dt.month etc