If you are using the Python stack for studying and applying machine learning, then the library that you will want to use for data analysis and data manipulation is Pandas.

This post gives you a quick introduction to the Pandas library and point you in the right direction for getting started.

Pandas for data analysis.
Photo by gzlu, some rights reserved.

Data Analysis In Python

The Python SciPy stack is a popular for scientific computing in general. It provides powerful libraries for handing gridded data (like NumPy) and plotting (like matplotlib). Until recently, a piece that had been missing from the suite was a good library for handling data.

Data, typically does not come in a form that is ready to used. A very large part of working on a data-driven problem like machine learning is data analysis and data munging.

• Data Analysis: This is using the tools like statistics and data visualization to better understand the problem by understanding the data.
• Data Munging: This is the process of transforming raw data into a form so that it is appropriate for your job, like data analysis or machine learning.

Traditionally, you had to cobble together your own tool-chain of scripts in Python to perform these tasks.

These days, if you search for data analysis in Python you can’t avoid learning about Pandas. It has quickly become the go-to library for data handling in Python.

What is Pandas?

Pandas is a Python library for data analysis and data manipulation. It adds the missing piece to the SciPy framework for handling data.

Pandas was create by Wes McKinney in 2008 primarily for quantitative financial work. As such it has a strong foundation in handling time series data and charting.

You use Pandas to load data into Python and perform your data analysis tasks. It is perfect for working with tabular data like data from a relational database or data from a spreadsheet.

Wes describes the vision of Pandas as to crate: the most powerful and flexible open source data analysis and manipulation tool available in any language.

An admirable mission that makes you want to support his cause, if only to make your own data analysis tasks easier.

Pandas Features

Pandas is a pleasure to use.

In my experience it is simple, elegant and intuitive. Having come from R, the idioms and operations are familiar and relevant.

Pandas is built on top of standard libraries in the SciPy stack. It uses NumPy for fast array handling, and provides convenient wrappers around some statistical operations from StatsModels and charting from Matplotlib.

There is a strong focus on time series given the libraries inception in the financial domain. It also has a strong focus on data frames for handling standard gridded data. Data handling is a core requirement of a library of this kind and speed has been made a priority. It is fast and provides data structures and operations like indexing and handling of sparsity.

Some important features to note include”

• Manipulation: moving columns, slicing, reshaping, merging, joining, filtering, and others.
• Time-series Handling: operations on date/times, resampling, moving windows and auto-alignment of datasets.
• Missing Data Handling: auto-exclude, drop, replace, interpolate missing values
• Group-by Operations: SQL like group by.
• Hierarchical Indexing: data structure level, powerful for efficiently organizing data by columns.
• Summary Statistics: Fast and powerful summary statistics of data.
• Visualization: Simplified access to plots on data structures, such as histograms, box plots, general plots and a scatter matrix.

Pandas is available under a permissive license (Simplified BSD) and can be easily installed along with the the rest of SciPy.

Pandas Resources

This has been a quick introduction to the Pandas library and there is more to learn. Install the library, grab a dataset and start to try things out. There is no better way to get started.

Visit the Pandas homepage and have a read of the library vision and features. You can also check-out the github page for the project.

A great place to start is the list of tutorials which includes links to cookbooks, lessons, and various notable IPython notebooks around the web.

Finally, for me, I live in the API documentation.

Papers

I find papers can give a good overview of an open source library, particularly in the Python and R ecosystems. Take a look at the following papers for a structured overview of what Pandas is all about.

Videos

There are a lot of great videos on YouTube of people demonstrating Pandas on their own data and at conferences.

A great starting point is Wes’ own 10-minute tour of pandas. Take a look. It’s a little time-series data heavy, but it’s a great and quick overview. You can also checkout his IPython notebook for this tour.

Books

Finally, Wes is the author of the definitive book on data analysis in Python. If you want to get serious, practice, but also consider grabbing the book. It’s called: Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython.