Scribd Logo
Upload

Welcome to Scribd!

  • 4 views

    ML m5_2

    Uploaded by

    puneethsp2004

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    ML m5_2

    Uploaded by

    puneethsp2004
    4 views24 pages

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    4 views24 pages

    ML m5_2

    Uploaded by

    puneethsp2004

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 24

    Cross-Validation and Resampling Methods

    • Cross-validation is a technique used to check how well a machine learning model will
    perform on new, unseen data. It’s especially useful when we don’t have a large dataset
    to work with.

    • It helps us estimate the model's accuracy and reliability by making the most of the
    available dataset

    • Why is cross-validation needed?

    • When building a machine learning model, we train it on a dataset and then test its
    performance on a separate dataset. However, splitting the data this way can lead to
    overfitting or under fitting, especially if the dataset is small. Cross-validation helps
    address this by using the data more effectively.
    • How does cross-validation work?

    • Divide the Dataset: The dataset is divided into multiple smaller parts, called
    folds.

    • Train and Test: The model is trained on some folds and tested on the remaining
    ones. This process is repeated multiple times:

    • In each iteration, a different fold is used as the test set, and the remaining
    folds are used for training.

    • Average the Results: The performance from all the test folds is averaged to get a
    final evaluation score.
    • Stratification is a technique used in data splitting to ensure that the proportions
    of different groups or classes in the dataset are maintained in every subset, such
    as training and validation sets. It ensures the data's structure remains balanced
    and representative of the original dataset.

    • There are several types of cross-validation techniques, each designed to handle


    different scenarios or data structures. Some of them are:

    • K-Fold Cross-Validation
    • 5 × 2 Cross-Validation

    • Bootstrapping
    K-Fold Cross-Validation

    •
    • There are 2 problems with this.

    • First, to keep the training set large, we allow validation sets that are small.

    • Second, the training sets overlap considerably, namely, any two training sets share K − 2 parts.

    • One extreme case of K-fold cross-validation is leave-one-out where given a dataset of N instances,
    only one instance is left out as the validation set (instance) and training uses the N − 1 instances.
    5 × 2 Cross-Validation
    Bootstrapping
    • Bootstrapping is a way to create multiple new datasets from a single original dataset .

    • Instead of splitting the data like in cross-validation, bootstrap creates new samples by randomly
    selecting data points from the original dataset with replacement.

    • Replacement refers to the process of randomly selecting data points from the original sample such that
    each data point can be chosen more than once.

    • For example, if your original dataset contains the points [A, B, C, D], a bootstrap sample might look like [B,
    B, D, A], where B is selected twice.

    • In the bootstrap, we sample N instances from a dataset of size N with replacement. The original dataset
    is used as the validation set.
    Measuring Classifier Performance
    • For classification, especially for two-class problems, a variety of
    measures has been proposed. There are four possible cases
    • This image illustrates the concept of Receiver Operating Characteristic (ROC) curves, which are
    commonly used to evaluate the performance of classifiers in machine learning.
    This image explains the concepts of precision and recall using Venn diagrams, which are
    commonly used in information retrieval and machine learning to evaluate the performance of
    classification models.

    You might also like