What is Cross-Validation in Machine Learning?

Cross-validation is a technique that is used for assessing the performance of the model in real-world scenarios after the dataset has been split into multiple parts. Here, some parts of the dataset are used for training the model, while the remaining parts of the dataset are used for testing other models. This will help you to avoid overfitting or underfitting. Cross-validation is also used to give a reliable and unbiased estimate of how accurately the model predicts outcomes on unseen data. You can also use it for tuning hyperparameters. 

In this blog, we are going to talk about the Cross-Validation technique in machine learning, its types, and its implementation in Python. So, let’s get started!

Table of Contents

• What is Cross-Validation?
• Types of Cross-Validation
• Implementation of Cross-Validation using k-Fold Cross-Validation in Python
• Advantages and Limitations of Cross-Validation
• Best Practices for Cross-Validation
• Conclusion

What is Cross-Validation?

Cross-validation is a technique that is used to evaluate the performance of the model on independent data. In cross-validation, the model divides the dataset into multiple subsets or “folds” instead of a single train-test split. The model is then trained on a combination of these folds and tested on the remaining folds. This process goes on to repeat several times until every data point gets a chance to be in the testing set. By calculating the average of the results, you can get a more stable and accurate estimate of the performance of the model. This is useful, especially while you are working with limited or unbalanced data.

Cross-validation in machine learning additionally enables better use of the data that is already accessible. Each observation in the dataset is used for both training and validation at some stage because the data is split up into numerous subsets. By doing this, the utility of data for model training and evaluation is maximized.

Types of Cross-Validation

There are different types of Cross-validation that can be used based on different scenarios. Let’s have a look at them:

1. K-Fold Cross-Validation

In the k-fold cross-validation, you have to split the dataset into k folds. Then the model is trained and evaluated k times, and the average of the performance metrics is calculated over k iterations. In each iteration, one fold is used for testing, and the rest k-1 folds are used for training. After the testing is done, you can calculate the average of the results. K-fold cross-validation is adaptable to a variety of datasets.

2. Stratified k-fold cross-validation

Stratified k-fold cross-validation is almost similar to k-fold cross–validation. The only difference is that it ensures that each group or fold represents the complete dataset accurately. Stratified k-fold is used particularly while dealing with biases and variances when there is an imbalance in the dataset or a class imbalance in the target variable.

3. Holdout method

It is the simplest form of cross-validation. It splits the dataset into two parts: a training set and a test set. This helps the model to be trained on the training set and evaluated on the test set. After that, the model is evaluated on the holdout data. Holdout data refers to a portion of the test data that is intentionally held out and left unseen during training. The holdout method turns out to be useful for large datasets.

4. Time series cross-validation

While you are working with time series data, you must be dependent on an advanced method called time series cross-validation, because the order of the data points is important. This approach helps to ensure that the model is always tested on future data, unlike the traditional cross-validation method. In each round, if you train the model on all the data up to a specific point in time and then test the model on the following data, it helps in preserving the structure of the model. This setup will help you make the evaluation of the model more realistic and useful for spotting patterns or trends over time.

5. Leave-one-out cross-validation

While using the LOOCV(leave-one-out cross-validation) method, you have to treat each data point in the set like its own validation set. This should be done while you train the model on the remaining data. Sp, if you have n total samples, you have to train and test the model n times. For k-fold cross-validation, it is a special case because here, k equals the total number of samples. This model provides a detailed evaluation of the model, which you can find useful while you are working with smaller datasets. You should avoid it for large datasets since the computation becomes too intensive.

Implementation of Cross-Validation using k-Fold Cross-Validation in Python

While working with various machine learning algorithms, k-fold cross-validation is used for evaluating the performance of the model. Given below are the steps for the implementation of k-fold cross-validation in Python.

Step 1: Importing the Required Libraries

In this step, you need to import the necessary libraries like pandas, numpy, sklearn for the building of model and its evaluation.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression

The above code is used to import the required libraries, the iris dataset, and set up the k-fold cross-validation, and also initializes the logistic regression model. This code does not generate an output because no function has been called by the code to perform the evaluation of the model or print any result.

Step 2: Load the Dataset

For the model evaluation, we will be training the model on the Iris dataset, which is appropriate for classification tasks.

# Now you have to load the iris dataset
iris = load_iris()
X = iris.data
y = iris.target

print(X[:5])
print(y[:5])

The code above is used to load the Iris dataset and then prints the first 5 features ( X[:5] ), and the corresponding target labels ( y[:5] ).

Step 3: Define the Model

The next step is to define the model that you want to evaluate. Here, we will be using the Logistic Regression model.

model = LogisticRegression(max_iter=200)

The above line of code helps in initializing the Logistic Regression model. Here, 200 iterations are allowed at max for the solver to converge during training. No output is generated after running the above code cell because it just defines the model.

Step 4: Set Up k-Fold Cross-Validation

In this step, you have to create a KFold object with k=5. This means that the dataset will split into 5 folds.

kfold = KFold(n_splits=5, shuffle=True, random_state=42)

The above line of code is used for creating a K-fold cross-validator. This splits the dataset into 5 shuffled folds, containing a random seed (42) for ensuring reproducibility. This code does not provide an output, it just prepares the cross-validation scheme.

Step 5: Evaluating the model using Cross-validation

This is the last and most important step of the implementation process. You have to evaluate the model by calculating its accuracy on each fold and then calculating its average.

results = cross_val_score(model, X, y, cv=kfold, scoring='accuracy')

print("Accuracy for each fold:", results)
print("Average Accuracy:", results.mean())

The above code evaluates the Logistic regression model. It uses 5-fold cross-validation on the iris dataset. After that, it prints the accuracy on each fold. At last, it prints the average accuracy across all the folds.

Advantages and Limitations of Cross-Validation

1. Advantages of Cross-Validation

• It helps in reducing the risk of overfitting.
• It provides an accurate estimate of the performance of the model.
• It uses data efficiently.

2. Limitation of Cross-Validation

• LOOCV (leave-one-out cross-validation) can be expensive for large datasets. This is because it requires training the model n times, where n is the total number of data points.
• Due to training the model repeatedly, it becomes very slow, especially when complex algorithms are used.
• LOOCV is not suitable for time series data by default. This is because it does not save the chronological order, which is important for the predictions based on time, unless they are modified appropriately.

Best Practices for Cross-Validation

• You should always shuffle the data unless you are dealing with time series.
• For classification problems, you should use stratified k-fold cross validation.
• You should look for all three metrics, like precision, recall, and F1 score, instead of depending on just one metric.
• You should always scale or normalize your data before cross-validation, especially while you are using algorithms that are sensitive to feature scales like SVM or logistic regression.
• You should always keep a separate test set that is untouched by cross-validation. This will help you to evaluate the true performance of the model on unseen data.

Conclusion

As we wrap up this blog, you have learnt that cross-validation is an important tool for any machine learning workflow. It gives you confidence that your model is going to perform well in the real world and not just on paper. By testing the model on various subsets of data, you can ensure that the model is not overfitting or underfitting. Mastering cross-validation will help you make more reliable and well-performing models. It also plays an important role in the selection of models and hyperparameter tuning. This helps you to develop robust and accurate models.

If you are interested in learning about these techniques, then we recommend you to check out our Data Science Course.