In my thinking, this warning message also covers cases where both y1 and y2 are empty (as in "test case: empty inputs"). At least it would motivate people to inspect their data and then they might find out if their data is empty.

However, we could also branch here and make a separate warning message for empty inputs.

In my thinking, this message also fits the cases when y1 and y2 only deal with one label (as in "test case: both inputs only have one label").

I would pick 0.0 as a future default (instead of -1.0 which is the worst score), because it is the least expressive of the scores, representing matching labels by chance.

If users would use cohen_kappa_score as part of their custom metric, that calculates the mean over several cohen_kappa_scores, 0.0 would be a neutral element like the "ignore" option that we have talked about in this comment: #29048 (comment)

The test function looks good to me overall—my comment is just a minor nitpick:

This way, we could also simplify the unpacking slightly:

    y1, y2, labels, weights, sample_weight = test_case
    y_1, y2 = np.array(y1), np.array(y2)

and only need one assert:

    assert check_equal(
        cohen_kappa_score(
            y1,
            y2,
            labels=labels,
            weights=weights,
            replace_undefined_by=replace_undefined_by,
        ),
        replace_undefined_by,
    )

Totally optional though—happy for you to resolve this if you’d prefer to keep it as is.

That's a good suggestion.

I have made this test leaner, I couldn't do without commenting every single test case though. 🤷 To help the next person looking at this see why these test cases trigger a division by zero.