Python Data Science Tutorial

• What Do You Understand from Python?
• Why Python for Data Science is the Go-To Language?
• Python as a “Leader”
• Why is Python preferred over others?
• R vs Python for Data Science
• How to install Python?
• Libraries in Python for Data Science
• Data Science with Python: Demo
• Companies that use Python for Data Science

Here’s a video from Intellipaat on Python for Data Science:

What Do You Understand from Python?

Python is a high-level, general-purpose programming language with an elegant syntax that allows programmers to focus more on problem-solving than on syntactical errors. One of the primary goals of Python developers is to keep the language fun to use. Python has gained massive buzz in the field of modern software development, infrastructure management, and, especially, Python for data science and artificial intelligence. Most recently, Python has risen to the top 3 list of the TIOBE index of language popularity.

Why Python for Data Science is the Go-To Language?

Python programming comes first when we think of data science. Python has rapidly gained popularity in the IT community as a simple yet feature-rich language powering anything from simple web applications to the IoT, game development, and even artificial intelligence.

Big data and data analytics are other sectors in which Python is currently making inroads. In this Python data science tutorial, let us find out why Python is used in big data.

So many programming languages provide the much-needed options to execute data science jobs. This has resulted in it being difficult to handpick a specific language.

But it is data that provides a peep into these languages that are making their way into the world of data science, i.e., nothing can be as compelling as the data itself unveiling the results of the comparison between different data science tools.

Python as a “Leader”

Python is one of the fastest-growing programming languages in the world, and it is quite easy to learn. Being a high-level programming language, Python is widely used in mobile app development, web development, software development, and the analysis and computing of numeric and scientific data.

Python can run on any platform—Windows, Linux, Macintosh, etc.

Why is Python preferred over others?

Codes in Python are written in a very natural style, which is why it is easy to read and understand.

Some of the features of Python that make it a popular language in data science applications are:

Easy to Learn

Python is for anyone aspiring to learn because of its ease of learning and understanding.

Python is a popular data science tool with 35 percent of data analysts using it. It follows R in popularity and is ahead of SQL and SAS. 

Scalability

Python is known to be an extremely scalable language when compared to other languages such as R. Python is also faster to use than MATLAB or Stata.

Python’s scalable nature lies in its flexibility during problem-solving situations because of which even YouTube has migrated to Python.

Python has come to be good for different usages in industries as many data scientists use this language to develop various types of applications successfully.

Availability of Data Science Libraries

The best answer to the question, Why Python for data science, is the availability of various libraries such as pandas, statsModels, NumPy, SciPy, and scikit-learn.

The constraints that developers faced a year ago are addressed by the Python community; the Python community helps address problems of a specific nature through robust solutions.

Python Community

One of the major factors behind the remarkable upsurge of Python in the industry is its ecosystem. Many volunteers are developing libraries in Python for Data Science as Python has extended its hands to the data science community, which, in turn, has led the way for creating the most modern tools and processing in Python. The community helps these Python aspirants with relevant solutions to their coding problems.

Graphics and Visualizations

Python’s diverse graphic and visualization options, including Matplotlib, seaborn, and pandas, offer invaluable tools for extracting insights from available data. These tools assist in illustrating trends, patterns, and correlations, helping in the comprehension of complex data structures, facilitating data-driven decision-making, and enhancing communication of findings within and beyond the organization.

These packages help in getting a good sense of data, creating charts, graphical plots, web-ready interactive plots, and much more.

R vs Python for Data Science

For almost a decade, researchers and developers have been debating the topic, R or Python for data science—which is a better language?

With the adoption of open-source technologies taking over the traditional, closed-source commercial technologies, Python and R have become extremely popular among data scientists and analysts.

But it has been noticed by Maruti Techlabs that “Python’s increase in the share over 2015 rose by 51% demonstrating its influence as a popular Data Science tool.”

How to install Python?

There are two ways to install Python:

• We can download Python directly from its website and install the needed individual components and libraries.
• Alternatively, we can also download and install a package, which comes with preinstalled libraries such as downloading Anaconda, Google Collab, Enthought Canopy Express

The second method is a more hassle-free installation and is ideal for beginners. However, one has to wait for the entire package to be upgraded, even if they just want the latest version of a single library. Unless there is advanced statistical research involved, this should not be a problem.

Google Colab, a cloud-based platform, is another excellent option for data science, offering collaboration and easy access to libraries. After installation, selecting a development environment is important.

The next step is choosing a development environment. Once Python is installed, there are various options to choose an environment. The following are the three most common options:

• IDLE (default environment)
• Terminal/Shell-based
• IPython Notebook

So let’s ahead in this python data science tutorial and understand the concepts of python libraries for data science.

Libraries in Python for Data Science

Python’s great readability and simplicity make it a popular, general-purpose programming language that enables developers to quickly build prototypes and applications. Its extensive library, especially with regard to data science and machine learning (NumPy, Pandas, and Scikit-learn), has greatly added to its popularity. Python is a preferred option for web development, scientific computing, automation, and artificial intelligence applications because of its versatility and scalability. The language has become the foundation of contemporary programming due to its ease of integration and compatibility with a wide range of platforms and systems. This has driven its widespread acceptance across numerous industries.

It has been reported that Python is used extensively by developers in the creation of games, standalone PCs, mobile applications, and other enterprise applications.

Python libraries simplify complex tasks and make data integration much easier with fewer codes in lesser time. Python has more than 137,000 libraries, which are very powerful and are vastly used to satisfy the requirements of customers and businesses. These libraries have helped scientists and developers analyze large amounts of data, generate insights, engage in critical decision-making, and much more.

The following are a few Python libraries that are widely used in the fields related to data science.

NumPy

NumPy, an extensive Python library, is essential for scientific computations and serves as an essential building block for data processing, especially in scientific and numerical calculations. It utilizes functions, N-dimensional array objects, and tools for integrating C/C++ and Fortran code, offering capabilities in mathematical concepts like linear algebra and random numbers. Numpy arrays are the backbone, aiming to facilitate quick and efficient operations on large datasets due to their uniform data format, setting them apart from regular Python lists and optimizing complex mathematical computations for improved memory efficiency and faster calculations.

SciPy

SciPy is another important library of Python for developers, researchers, and data scientists out there. It includes optimizations, statistics, linear algebra, and integration packages for computation. It can be of great help for someone who has just started a career in data science to guide them through numerical computations.

Pandas

Pandas is the most powerful open-source library of Python for data manipulation. It is known as the Python Data Analysis Library. It is developed over the NumPy package. DataFrames are considered the most used data structures in Python that help in handling and storing data from tables by performing manipulations over rows and columns. Pandas are very useful in merging, reshaping, aggregating, splitting, and selecting data. Moreover, through its versatile data structures and functionalities, significantly streamline Exploratory Data Analysis (EDA) by facilitating easy data manipulation, cleaning, and analysis for insightful decision-making.

Scikit-Learn

Scikit-Learn is a collection of tools for performing mining-related tasks and data analysis. Its foundation is built over SciPy, NumPy, and Matplotlib. It consists of classification models, regression analysis, image recognition, data reduction methods, model selection and tuning, and many other things.

Statsmodels

statsmodels for statistical modeling. statsmodels is a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available for different types of data and each estimator.

Blaze

Blaze for extending the capability of Numpy and Pandas to distributed and streaming datasets. Blaze can be used to access data from a multitude of sources including bcolz, MongoDB, SQLAlchemy, Apache Spark, PyTables, etc. Together with Bokeh, Blaze can act as a very powerful tool for creating effective visualizations and dashboards on large chunks of data.

Scrapy

Scrapy for web crawling. Scrapy is a very useful framework for getting specific patterns of data. It can start at a website’s home URL and then dig through web pages on the website to gather information.

SymPy

SymPy for symbolic computation. SymPy has wide-ranging capabilities from basic symbolic arithmetic to calculus, algebra, discrete mathematics, and quantum physics. Another useful feature of SymPy is its capability of formatting the result of the computations as LaTeX code.

Matplotlib

Matplotlib is a popular plotting library of Python that data scientists extensively use for designing numerous figures in multiple formats depending on their compatibility across their respective platforms. For example, with Matplotlib, you can create your own scatter plots, histograms, bar charts, and so on. It provides good quality 2D plotting and basic 3D plotting with limited usage.

import matplotlib.pyplot as plt

# Sample data
x_values = [1, 2, 3, 4, 5]
y_values = [2, 4, 3, 5, 6]

# Creating a scatter plot
plt.figure(figsize=(8, 6))  # Set the figure size
plt.scatter(x_values, y_values, color='blue', label='Scatter Plot')

# Adding labels and title
plt.xlabel('X-axis Label')
plt.ylabel('Y-axis Label')
plt.title('Scatter Plot')

# Adding a legend
plt.legend()

# Show the plot
plt.show()

Seaborn

Seaborn for statistical data visualization. Seaborn is a library for making attractive and informative statistical graphics in Python. Based on Matplotlib, Seaborn aims to make visualization a central part of exploring and understanding data.

import seaborn as sns
import plotly.express as px
import pandas as pd
import numpy as np

# Generating large random data
np.random.seed(42)
num_points = 100
x_values = np.random.rand(num_points)
y_values = np.random.rand(num_points)
colors = np.random.rand(num_points)
sizes = np.random.rand(num_points) * 10

data = pd.DataFrame({
    'X': x_values,
    'Y': y_values,
    'Colors': colors,
    'Sizes': sizes
})

# Using Seaborn for styling
sns.set(style="whitegrid")

# Creating an interactive scatter plot using Plotly Express
fig = px.scatter(data, x='X', y='Y', size='Sizes', color='Colors', color_continuous_scale='Viridis',
                 labels={'X': 'X-axis', 'Y': 'Y-axis'},
                 title='Interactive Scatter Plot with Seaborn Styling')
fig.show()

Bokeh

Bokeh for creating interactive plots, dashboards, and data applications on modern web browsers. Bokeh empowers the user to generate elegant and concise graphics in the style of D3.js. Moreover, it has the capability of high-performance interactivity over very large or streaming datasets.

To achieve highly interactive plots, Bokeh is more commonly used compared to Matplotlib or Seaborn.

Requests

Requests for accessing the web. It works similar to the standard Python library urllib2 but is much easier to code. You will find subtle differences between Requests and urllib2, but for beginners, Requests may be more convenient.

Some additional libraries that one may need are:

• os for the operating system and file operations
• NetworkX and graph for graph-based data manipulations
• Regular Expression for finding patterns in text data
• Beautiful Soup for performing web scraping by extracting information from just a single web page in a run

Data Science with Python: Demo

Problem Statement

You are a data scientist at a telecom company Neo whose customers are switching over to its competitors. You have to analyze the data of your company and find insights and stop your customers from switching over to other telecom companies

Dataset

This is a snapshot of the dataset that you will be working on:

                                                                                               Python for Data Science

Tasks to be Done

• Data Manipulation: Extracting individual rows and columns from the dataset and finding interesting patterns
• Data Visualization: Understanding individual columns from the dataset by visualizations
• Model Building: Building a decision-tree model

Data Manipulation

We will start by loading the required packages:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

Now, let us load the customer_churn dataset:

#reading file
customer_churn = pd.read_csv("customer_churn.csv")

Glancing at the first few rows of the dataset:

#Looking at the first few rows
customer_churn.head()

Python for Data Science

Extracting the 5th column from the entire dataset:

#Extracting 5th column
customer_5=customer_churn.iloc[:,4] 
customer_5.head()

Extracting male senior citizens with payment method -> Electronic check:

senior_male_electronic=customer_churn[(customer_churn['gender']=='Male') & (customer_churn['SeniorCitizen']==1) & (customer_churn['PaymentMethod']=='Electronic check')]
senior_male_electronic.head()

Data Visualization

Making a bar-plot for the distribution of the “Internet Service” column:

plt.bar(customer_churn['InternetService'].value_counts().keys().tolist(),customer_churn['InternetService'].value_counts().tolist(),color='orange')
plt.xlabel('Categories of Internet Service')
plt.ylabel('Count of categories')
plt.title('Distribution of Internet Service')

Making a histogram for the distribution of the “tenure” column:

plt.hist(customer_churn['tenure'],color='green',bins=30)
plt.title('Distribution of tenure')

Model Building

Let us build a decision-tree model on top of the “customer_churn” dataset, where “Churn” is the dependent variable and “tenure” is the independent variable.

We will start by extracting “Churn” and “tenure” from the original dataframe:

x=pd.DataFrame(customer_churn['tenure'])
y=customer_churn['Churn']

Now, let us divide our data into “train” and “test” sets:

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.20)

“train_test_split” splits the dataset ‘x’ and ‘y’ into training and testing sets respectively. The ‘test_size=0.20’ parameter assign 20% of the data to the testing set, whereas the remaining 80% is designated for training the model. The ‘x_test’ and ‘y_test’ are for testing the model’s performance and the ‘x_train’ and ‘y_train’ are for training.

We will import the decision-tree classifier and fit the model on top of the train set:

from sklearn.tree import DecisionTreeClassifier 
classifier = DecisionTreeClassifier() 
classifier.fit(x_train, y_train)

The “fit” function within the DecisionTreeClassifier() trains the model by learning patterns and relationships between the features (x_train) and the corresponding target labels (y_train) provided in the training dataset, enabling the classifier to make predictions based on this learned information.

Now that we have fit the model on the “train” set, it is time to predict the values on the “test” set:

y_pred = classifier.predict(x_test)

The “predict” function uses the trained classifier to forecast or predict the target labels for the test dataset (x_test), based on the learned patterns from the training data, allowing evaluation of the model’s performance by comparing predicted values (y_pred) against actual labels.

Let us now go ahead and calculate the accuracy:

from sklearn.metrics import classification_report, confusion_matrix,accuracy_score

The “confusion_matrix” provides a tabular summary of model performance, showing true positive, false positive, true negative, and false negative results. Whereas, the “accuracy_score” calculates the proportion of correct predictions to the total predictions, offering an overall measure of model accuracy.

print(confusion_matrix(y_test, y_pred)) 
print(accuracy_score(y_test, y_pred))

We see that we get an accuracy of 73.88% for this decision-tree model.

Companies that Use Python for Data Science

Instagram

Instagram has about 400 million daily active users who share more than 95 million photos and videos.

Instagram has recently moved to Python 3, and the main reason why Instagram chose Python was its simplicity and popularity.

Instagram claims to have considered different languages over Python but did not get any significant performance improvement.

Spotify

Spotify trusts Python and uses it for back-end services, as well as for data analysis.

Spotify claims that the speed of development is its priority, and that is the reason why Spotify uses Python to build its music streaming service as Python meets Spotify’s development speed expectations.

For data analysis, Spotify uses Hadoop with Python to process large amounts of data to polish its services.

Amazon

Amazon analyzes customers’ buying habits and searches patterns to provide them with accurate recommendations. It is possible due to their Python machine learning engine, which interacts with Hadoop, the company’s database. They work in conjunction to achieve maximum efficiency and accuracy in providing recommendations to customers.

Amazon prefers Python because it is popular, scalable, and appropriate for dealing with big data.

Facebook

Facebook deals with large amounts of data, including tons of images, and it uses Python to process the images.

Facebook decided to use Python for its back-end applications connected with image processing, such as image resizing, because of its simplicity and ease of development.

SurveyMonkey

SurveyMonkey is one of the largest survey companies in the world. It processes more than one million survey responses daily.

Initially, SurveyMonkey’s web app was built on .NET along with C#. There weren’t any issues with the smoothness of the system, but it got relatively slow in testing while deploying new features.

SurveyMonkey rewrote its app in Python and broke the main features into several separate services and these services were communicated through the web APIs. This allowed SurveyMonkey to implement features on smaller codebases that can be managed more easily.

SurveyMonkey chose Python because of its simplicity, availability of tons of libraries to build web apps faster, availability of tools that facilitate deployment, unit testing, etc.

Conclusion

Python is a great tool and is becoming an increasingly popular language among data scientists as it is easy to learn and integrates well with other databases and tools such as Spark and Hadoop. It also has great computational intensity and powerful python data analytics libraries.  

So, learn Python to perform the full life cycle of any data science project. It includes reading, analyzing, visualizing, and making predictions. From this Python data science tutorial, you would have learned why python is preferred over any language and python libraries for data science.