Implementing Cross-Validation in R with the 'caret' Package

    Cross-validation is a powerful technique used to evaluate machine learning models with different subsets of training data. It helps enhance model accuracy and prevent overfitting, ensuring robust estimations. 

    In this tutorial, we'll explore how to implement cross-validation in R using the 'caret' package. We will use the famous Iris dataset as our test data.

  

Load the Necessary Libraries and the Iris Dataset

    First, we need to load the 'caret' package, which contains useful functions for cross-validation. Additionally, we'll load the Iris dataset, a popular dataset commonly used for classification tasks.

 

# Load the caret package for cross-validation

library(caret)

# Load the Iris dataset from the datasets package

data(iris)

 

Define Cross-Validation Parameters with 'trainControl'

    To perform cross-validation, we need to set up the parameters using the 'trainControl' function. In this example, we will use 10-fold cross-validation, meaning the data will be divided into 10 subsets, and the model will be trained and tested on each subset.

 

# Define cross-validation settings with 10 folds

tc <- trainControl(method = "cv", number = 10)

 

Fit the Model Using the 'train' Function

    Now, we are ready to train our model using the 'train' function. For this tutorial, we will use the random forest algorithm as our training method.

 

# Fit the model using random forest algorithm

fit <- train(Species ~ .,

data = iris,

method = "rf",

trControl = tc,

metric = "Accuracy")

 

Review the Cross-Validation Results

After fitting the model, we can review the cross-validation results to see how well our model performed. The 'print' function will display the accuracy and kappa values obtained for different values of 'mtry', which is a parameter for the random forest algorithm. The highest accuracy value will be selected as the optimal model.

 

# Print the cross-validation results

print(fit)

Random Forest

150 samples
4 predictor
3 classes: 'setosa', 'versicolor', 'virginica'

No pre-processing
Resampling: Cross-Validated (10 fold)
Summary of sample sizes: 135, 135, 135, 135, 135, 135, ...
Resampling results across tuning parameters:

mtry Accuracy Kappa
2 0.9466667 0.92
3 0.9466667 0.92
4 0.9400000 0.91

Accuracy was used to select the optimal model using
the largest value.
The final value used for the model was mtry = 2.

 

Predict Iris Data and Evaluate the Model

Finally, we use the trained model to predict the species of the Iris data, and we evaluate our model using a confusion matrix. The confusion matrix shows the number of correct and incorrect predictions made by the model.

 

# Predict Iris data using the trained model

pred <- predict(fit, iris[, -5])

# Evaluate the model using a confusion matrix

confusionMatrix(iris[, 5], pred)

Confusion Matrix and Statistics

Reference

Prediction setosa versicolor virginica

setosa 50 0 0

versicolor 0 50 0

virginica 0 0 50

Overall Statistics

Accuracy : 1

95% CI : (0.9757, 1)

No Information Rate : 0.3333

P-Value [Acc > NIR] : < 2.2e-16

Kappa : 1

Mcnemar's Test P-Value : NA

Statistics by Class:

Class: setosa Class: versicolor

Sensitivity 1.0000 1.0000

Specificity 1.0000 1.0000

Pos Pred Value 1.0000 1.0000

Neg Pred Value 1.0000 1.0000

Prevalence 0.3333 0.3333

Detection Rate 0.3333 0.3333

Detection Prevalence 0.3333 0.3333

Balanced Accuracy 1.0000 1.0000

Class: virginica

Sensitivity 1.0000

Specificity 1.0000

Pos Pred Value 1.0000

Neg Pred Value 1.0000

Prevalence 0.3333

Detection Rate 0.3333

Detection Prevalence 0.3333

Balanced Accuracy 1.0000

 

 

Conclusion

    Cross-validation is an essential technique to ensure that our machine learning models are robust and capable of generalizing to unseen data. By following these steps and implementing cross-validation in R using the 'caret' package, we can confidently assess our model's performance and make better decisions when deploying our models for real-world applications. With a solid understanding of cross-validation, we can build more reliable and accurate machine learning models. 

 

Source code listing

 

 

# Load the caret package for cross-validation

library(caret)

# Load the Iris dataset from the datasets package

data(iris)

# Define cross-validation settings with 10 folds

tc <- trainControl(method = "cv", number = 10)

# Fit the model using random forest algorithm

fit <- train(Species ~ .,

data = iris,

method = "rf",

trControl = tc,

metric = "Accuracy")

# Print the cross-validation results

print(fit)

# Predict Iris data using the trained model

pred <- predict(fit, iris[, -5])

# Evaluate the model using a confusion matrix

confusionMatrix(iris[, 5], pred)