Outlier Detection

Ensuring data accuracy is important for making reliable decisions, but sometimes datasets contain unusual values called outliers. An outlier is like finding an odd number in a set of even numbers, i.e., it stands out from the rest. They can appear due to errors, natural variation, or rare events, and they may sometimes provide useful insights, such as detecting fraud. In this article, we will learn what outliers are, why detecting them is important, the different types, common detection methods, and real-world applications.

What is an Outlier?

An outlier is a data point that is different from the rest of the data because its value is unusually high or low compared to other data points or observations. Outliers can occur in any dataset in the form of numbers, measurements, or categories. They can mislead the data analysis, or sometimes can reveal interesting points also.

For example, in a classroom, most of the students have heights between 150cm to 180cm, and one student has a height of 220cm. Then the person with the longest height is an outlier because their height is quite different from the others.

Set Yourself Apart in Data Analysis

Analyze Better with Data Analysis Skills

Explore Program

Why is Outlier Detection Important?

Outlier detection is the process of finding the unusual points or outliers in the dataset. It is important because outliers can affect the results of your analysis.

• Impact on Statistical Analysis: Many statistical methods follow a normal distribution, but outliers can have a big impact on calculating other values like mean, standard deviation, and so on.
• Better Model Accuracy in Machine Learning: The machine learning models are very sensitive to data; if an outlier occurs in it, they can predict the wrong result.
• Identifying Errors and Noise: Outliers indicate mistakes that can occur in data entry or measurement, like recording 5000cm instead of 50cm.
• Understanding Real Data Patterns: Not all outliers are bad; they can sometimes be valuable insights, and help you to understand patterns, trends, and exceptional cases in your data. For example, in sales, most of the products are sold 50 to 100 per month, but suddenly a case arises of 500 products in a month. This may indicate a good marketing campaign or a new trend for the month.

Types of Outliers

An outlier can occur in many categories. Some of them are as follows:

1. Global Outliers (Point Outliers)

A point outlier is a single data point in a dataset present far away from the majority of the data. It is also called global because it stands out compared to the rest of the data. It is mainly an extremely high or low value, which is easy to spot visually, and often affects the value of mean, variance, and machine learning models. It can be caused by measurement or data entry errors.

2. Contextual Outliers

Contextual outliers are the set of data points that are unusual only in a specific context, i.e., the value can be normal in one situation but an outlier in another situation. They depend on contextual factors, like time, season, or location. It can sometimes impact the detection of abnormal trends or predictive analysis. For example, 35°C in summer is normal, but 35°C in winter is an outlier.

3. Collective Outliers

Collective outliers are a group of data points that together are different from the expected pattern, even if the individual data points are not extremely different. They are identified by looking at the overall pattern rather than single values, i.e., individual points may seem to be normal, but together they are different from the rest of the dataset.

Step-by-Step Outlier Detection

Outlier detection is a systematic process that helps you identify and handle the unusual data points appropriately. Below are the steps to detect an outlier in a dataset.

1. Understand the Data: You cannot detect the outliers if you don’t know your data; hence, understand your data, which will help you decide which points are unusual and which have normal variation. You have to check the data type of your data, look for missing values, and calculate basic statistics, like mean, median, and so on.
2. Visualize the Data: Visualizing the data makes outliers obvious and can be easily spotted by looking at the figure. Some of the common visualization techniques are boxplot, scatterplot, histogram, etc.
3. Choose a Method: There are different methods for detecting outliers, and the choice of the method you are using depends on various factors like data type, size, and context. Some of the common methods are statistical, visualization, and machine learning methods.
4. Detect Outliers: Once you understand the data and choose a method, the next step is to actually find which points are unusual.
5. Handle Outliers: After outlier detection, decide what to do with them because handling them depends on the cause and effect. You can remove outliers, transform your data, or keep them if they are meaningful.

Outlier Detection Methods

Outlier detection can be done using different approaches, some of which are as follows:

Statistical Methods

Statistical Methods are the methods that rely on mathematical measures like mean, standard deviation, and percentiles to find outliers. Some of the popular methods are:

A. Z-Score Method

The Z-score method is based on the concept of standard deviations, i.e., if the data point is very far away from the mean, it can be an outlier. This method works well for the data that is normally distributed, where the data lies

• The mean (µ) tells us the central value of the data.
• The standard deviation (σ) tells us how spread out the data is.

The Z-score measures the distance of a data point from the mean in terms of standard deviations, and

• if the Z-score is close to 0, it means that the data point is near the mean.
• If the Z-score is +2, it means that the data point is 2 standard deviations above the mean.
• If the Z-score is -3, it means that the data point is 3 standard deviations below the mean.

Z=(X−μ​)/σ

Where:

• X = data point
• μ = mean of the data
• σ = standard deviation

B. IQR Method (Interquartile Range)

In IQR, the method uses percentiles to detect outliers, instead of using the mean and standard deviation. It uses

• The median (Q2) divides the dataset into two equal halves.
• The first quartile (Q1) is the 25th percentile, the value below which 25% of the data lies.
• The third quartile (Q3) is the 75th percentile, the value below which 75% of the data lies.
• The Interquartile Range (IQR) is the difference between Q3 and Q1, i.e., IQR=Q3-Q1

Any data point that lies below (Q1 − 1.5 × IQR) or above Q3 + 1.5 × IQR is considered an outlier.

Machine Learning Methods

Machine learning methods use different algorithms that learn the structure of the data and then detect points that don’t fit the pattern. This method is very useful when the dataset is very large and complex, due to which the outliers are not easy to detect by the help of simple statistics. Now, let’s discuss some of the popular machine learning models.

A. Isolation Forest

The Isolation Forest method is based on the principle that outliers are rare and different, and can be separated from the rest of the data faster as compared to the normal data points. It builds many random decision trees and splits each tree with a value. The normal points in the dense region require more splits, and outliers get isolated very quickly because they differ strongly from the majority.

It starts with building random trees, and for each data point, it measures the average path length across all the trees. If there is a shorter path, it has more chances to be an outlier as compared to the longer path, which can be more likely to be a normal data point.

For example, most of the customers spend an amount of money approximately rupees 50 to 500. But one customer spends an amount of 10000. During random splitting in Isolation Forest, the point with a value of 10,000 will be isolated faster than typical points.

B. Local Outlier Factor (LOF)

LOF detects the outlier by comparing the density of a point to the density of the points of its neighbors. If the point lies in a region that is much denser than its neighbor, it will be treated as an outlier, whereas if a point has a similar density as its neighbor, it will be treated as a normal data point.

It works on the principle of k-nearest neighbors, where it finds the distance and calculates the local density of each point and how close it is to the point. Then it compares the density of each point with its neighbors. If a point’s density is much lower, it will assign a high LOF score (likely outlier).

For example, suppose you have GPS data of people walking. Most people walk in dense groups, but one person is slightly away from the group of other people, not extremely far, still it will be flagged as an outlier.

C. DBSCAN (Density-Based Clustering)

DBSCAN is a clustering-based algorithm that detects the outliers as noise. It groups the points that are close and dense, and then points in low-density regions are labeled as outliers(noise). It has the parameters

• ε (epsilon): The maximum distance between two points to be considered neighbors.
• MinPts: The minimum number of points required to form a dense cluster.

For example, in a map of restaurants, the majority of the restaurants are present in cities, and a few restaurants are far away from the city. The DBSCAN marks those isolated restaurants as outliers.

Visualization Techniques

Outlier detection can often be done visually before applying complex statistical or machine learning methods. Visualization helps in understanding the data distribution, spotting unusual patterns, and communicating results effectively. Some of the most common visualization techniques are:

A. Boxplot

A boxplot displays the distribution of data based on quartiles, with each box representing the interquartile range (IQR). Points outside the boxplot are considered outliers. They are best for data with single data because they quickly identify outliers in numerical columns.

B. Scatter Plot

A scatter plot illustrates the relationship between two variables, where outliers are represented by points that are far away from the general cluster. They are best for data with two variables because they detect the outliers when data points follow a clear trend.

C. Histogram

A histogram shows the frequency distribution of values, and outliers are easily visible if they show up as very low-frequency bars far from the main group. It is best when you have to detect the outliers for extreme values visually.

Tools and Libraries for Outlier Detection

Outlier detection can be performed using a variety of tools simple visualization packages to advanced machine learning libraries. Some of them are as follows

1. Python Libraries

Python has a vast range of libraries for data analysis and machine learning, which can be used for outlier detection. Some of them are as follows

A. NumPy & SciPy

Numpy is a Python core library mainly used for numerical computing, and Scipy is built on it to provide statistical methods. They provide simple mathematical calculations, such as mean, standard deviation, which are the first step to detect the unusual values.

Example:

Output:

In the above code, the mean and standard deviation functions of the numpy library are used to calculate z-scores.

B. Pandas

Pandas library is used for data manipulation and analysis, in which the outliers can be easily detected through group-based statistics, making it very easy. It integrates well with other libraries and also provides built-in functions like .quantile() to apply IQR-based outlier detection.

Example:

Output:

In the above code, the pandas library is used for outlier detection by using the Series and quantile functions.

C. Scikit-learn (sklearn)

Scikit-learn is the most widely used machine learning library, which provides advanced algorithms for outlier detection, like Isolation Forest. It works on both simple and advanced machine learning approaches.

Example:

Output:

In the above code, the IsolationForest module is used from the sklearn.ensemble library to calculate the outliers. In the code, the value 1 is considered normal, and -1 is considered an outlier.

2. R Libraries

R is the most popular language for statistics and data analysis, and has a rich ecosystem of libraries that can be used for outlier detection. R can be used to apply both classical statistical tests and modern machine learning approaches to spot unusual data points. Some of the common R libraries for outlier detection are as follows:

• outliers
• mvoutlier
• DMwR (Data Mining with R)
• Robustbase
• caret (Classification and Regression Training)
• ggplot2

3. Specialized Software

Specialized software makes the data exploration and analysis easy for non-programmers, too. Unlike the libraries in Python or R, which provide a drag-and-drop option, this makes the task easier and helps many domains, such as business, finance, healthcare, and research, where quick insights are needed. Some of the popular software are:

• Excel
• Google Sheets
• Tableau
• Power BI
• RapidMiner
• KNIME

Challenges in Outlier Detection

Outlier detection is a complex part of data analysis, and it can have many different challenges, such as:

1. Ambiguity in Defining Outliers: There is no universal rule to accept data as an outlier, because they depend on various factors, like context, data distribution, and domain knowledge. For example, a salary of 1L can be an outlier in a small company, but normal in an MNC.

2. High-Dimensional Data: Datasets with many features, some characteristics become less important and behave differently in outlier detection. For example, a patient’s height and weight may seem normal individually, but the BMI calculated from them may be abnormal.

3. Dynamic or Evolving Data: In many real-world applications, data changes over time; this phenomenon is called concept drift, which means that the statistical properties of the data are not fixed. Due to the data drift, there is a possibility that the data that seems unusual today may be perfectly normal tomorrow, and vice versa.

4. Noise vs True Outliers: When detecting outliers, it’s important to distinguish between the noise and true outliers because treating them the same way can cause problems. The noise is a random, isolated point present far from all data, whereas outliers are rare but meaningful observations.

5. Imbalanced Data: Outliers are usually rare compared to normal data, and this imbalance makes detection challenging, especially for machine learning models that may treat outliers as insignificant.

Applications of Outlier Detection

Outlier detection has many applications, some of which are as follows:

1. Financial Fraud Detection: In the financial sector, the fraudulent transactions often appear as outliers when compared to normal spending behavior. For example, a customer normally spends 5000 to 10000 monthly, but suddenly makes a transaction of ₹1,00,000 in a foreign country, this unusual activity is flagged as a potential fraud.

2. Cybersecurity: In cybersecurity, most systems follow a normal pattern, but when an attacker tries to exploit a system, it creates an abnormal activity in logs, network traffic, or user behavior. These abnormal activities are detected as outliers.

3. AI/ML Modeling: Machine learning models depend on the patterns present in the data to make the predictions, but outliers interfere with the predictions as they do not follow any pattern. Hence, detecting and addressing outliers is important for building robust AI/ML models.

4. Healthcare and Medical Diagnostics: Hospitals collect patient data such as temperature and heart rate. Some unusual values may be outliers and can indicate either a potential health issue or a data error.

5. Anomaly Detection in Big Data and Cloud Systems: Cloud systems and IT services produce huge amounts of data every second (like CPU usage, memory usage, network activity), and humans cannot watch all this manually. Outlier detection helps to automatically spot the unusual behavior in these systems.

Take Your Data Analysis Skills to the Next Level

Gain Insights with Our Data Analysis Training

Explore Program

Conclusion

From the above article, we learned that an outlier is a data point that is different from the rest of the data, having the three main categories, point, contextual, and collective outliers. Outlier detection is important to understand the real data patterns. It can be detected in many ways, like simple statistical analysis, visualization, and machine learning models. Many different tools can be used for outlier detection, like Python and R libraries, or some specialized software like Excel, Tableau, or Power BI. It is an important concept and has a vast application in various domains, like cybersecurity, healthcare, AI/ML models, and so on.

Useful Resources

• Data Science Course Syllabus
• How to learn Data Science from Scratch?
• How to Become a Data Scientist?

Outlier Detection – FAQs