The way you are evaluating is equivalent to using LeaveOneOut with cross_val_predict and getting the aggregated predictions to compute the evaluation metric.

It is known to not be appropriate to evaluate a model:

Warning: Note on inappropriate usage of cross_val_predict

The result of cross_val_predict may be different from those obtained using cross_val_score as the elements are grouped in different ways. The function cross_val_score takes an average over cross-validation folds, whereas cross_val_predict simply returns the labels (or probabilities) from several distinct models undistinguished. Thus, cross_val_predict is not an appropriate measure of generalization error.

So I think that what you observe here is this problem.

Hi, thanks for your fast reply. However, I don't think this is the issue. If I read cross_val_predict, the problem is more that the original y_labels might not line-up with the outputted ones anymore. If you look in my code, why is it in inappropiate to generate predictions for X_test, as long as we save y_test to make the comparison fair later? Furthermore, that wouldn't explain why the issue is only there with the SVM with probability=True, but is not there with other models such as LogisticRegression

To verify this, I rewrote the code (lot simpler), to this (which should be better right?):

from sklearn import svm
import numpy as np
from sklearn.model_selection import LeaveOneOut, cross_val_score
from tqdm import tqdm
import pandas as pd

# set up parameters for the experiments with fake data
N = 20
N_FEATURES = 50
N_TRIALS = 500

scores = []
for z in tqdm(range(N_TRIALS)):
    # generate fake data
    X = np.random.normal(0, 1, size=(N, N_FEATURES))
    y = np.random.binomial(1, 0.5, size=N)

    # for the first 20% of trials, we want to make sure that the y_labels are perfectly balanced, i.e. the mean=0.5
    # we do this, because our hypothesis is that something weird happens in this setting
    if z < int(N_TRIALS/5):
        y = np.array([0, 1] * int(N/2))
        y = np.random.permutation(y)

    # initialize the SVC
    clf = svm.SVC(probability=True, C=0.01)
    
    # calculate scores using cross_val_score
    nested_score = cross_val_score(clf, X=X, y=y, cv=LeaveOneOut(), scoring='neg_brier_score')
    scores.append([np.mean(y),
                   np.mean(nested_score)])

# put the scores in a pandas DataFrame for nicer inspection
df_scores = pd.DataFrame(scores)
df_scores.columns = ['y_label', 'brier']
df_scores['y_0.5'] = df_scores['y_label'] == 0.5
df_scores = df_scores.groupby(['y_0.5']).mean()
print(df_scores)

but the issue still persists. And only if the mean of y_label is exactly 0.5. (i.e. the mean Brier of the random data with y_labels that are not balanced is 0.30, which is expected with random data, but for perfectly balanced classes, the mean is 0.16, and the highest error in these 500 trials is only 0.19.)

Could it be something with the internal cross-validation of the SVM because of probability=True?

This is strange. When replacing

clf = svm.SVC(probability=True, C=0.01)

with

clf = CalibratedClassifierCV(svm.SVC(C=0.01), cv=5)

the results are as one would expect, while if I understand the documentation correctly, this is what SVC(probability=True) does under the hood?

This uses the same Platt method. However, CalibratedClassifierCV is implemented by scikit-learn while the Platt calibration for the SVM is done in the libsvm if I am not wrong. I will have a look.

I think the difference is that one of them (I think scikit-learn?) averages the cross-validated results while libsvm refits the svm on the whole data and uses the fitted sigmoid model? Though that wouldn't explain the mismatch I think? libsvm's platt scaling had some interesting edge cases I think, but I don't remember which one would explain this behavior.

Also see #16145.

My confusion on the same issue four years ago can be found here: #13662 (comment)

I think the conclusion there was that CalibratedClassifierCV uses stratified sampling and libsvm does not, and the LOO is indeed the culprit here.

I think the difference is that one of them (I think scikit-learn?) averages the cross-validated results while libsvm refits the svm on the whole data and uses the fitted sigmoid model? Though that wouldn't explain the mismatch I think? libsvm's platt scaling had some interesting edge cases I think, but I don't remember which one would explain this behavior.

Also see #16145.

My confusion on the same issue four years ago can be found here: #13662 (comment)

thanks for these links! While these are definitely useful and closely related, they do not seem to mention this specific issue raised here though (but I might be missing/misunderstanding something of course).

What I think happens at a higher level, is that there is something wrong/strange in the libsvm's platt scaling in these specific circumstances (low C, LOO-CV, balanced y). I think the balanced y and LOO-CV are needed, because probably the fitted sigmoid model of libsvm is somehow biased, and because of the combination of the balanced y and LOO-CV, if the bias is against the missing label in y_test, somehow, you end up predicting very well, as is happening here.

I was wondering how it is possible that with random data, that is not passed to SVM(probability=True), the SVM still correctly predicts. The only thing I can think of is the scenario above. Let me clarify it with an example:

• y=np.array([0,0,0,0,0,1,1,1,1,1]) (i.e. 5 positive classes, 5 negative classes)
• we run LOO-CV, and during the first iteration, y_test is therefore 0
• y_train now has 4 negative classes, 5 positive classes
• I think something goes wrong in the libsvm Platt's scaling, that biases its predictions somewhat to 0
• We correctly predict y_test as being closer to 0 than a random prediction
• etc etc

Hope this makes sense?

EDIT: and maybe good to emphasize, with higher values for C (i.e. 10 or 1000) the strange results do not happen and the predictions are random

I think this is closely related to an issue that AutoGluon has seen in their stacking:
autogluon/autogluon#2779

Essentially calibration is stacking, and we're facing the same information leakage here.