ENH Array API support for f1_score and multilabel_confusion_matrix #27369

OmarManzoor · 2023-09-14T10:33:56Z

Reference Issues/PRs

Towards #26024

What does this implement/fix? Explain your changes.

Adds array api support for f1_score and the functions related to it.
Converts the relevant metric values to a float so that a scalar is returned. Ref: RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar? #27339

Any other comments?

CC: @ogrisel @betatim

sklearn/preprocessing/_label.py

github-actions · 2023-09-14T10:35:44Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 88a48e0. Link to the linter CI: here}

sklearn/utils/_array_api.py

OmarManzoor · 2024-05-20T13:10:25Z

@ogrisel Could you kindly have a look at this PR?

sklearn/metrics/_classification.py

ogrisel

Overall this looks good to me. I am surprised it works without being very specific about device and dtypes, but as long as the tests (and they do), I am happy.

sklearn/metrics/_classification.py

ogrisel · 2024-05-24T14:31:05Z

sklearn/metrics/_classification.py

-        tp = np.array(tp)
-        fp = np.array(fp)
-        fn = np.array(fn)
+        sample_weight = xp.asarray(sample_weight)


We should probably make sure that it matches the device of the inputs, no? It's curious that existing tests do not fail with PyTorch and MPS device (or cuda devices).

I am also wondering of whether we should convert to a specific dtype. However looking at the tests I never see any case where we pass non-integer sample weights. And even for integer weights, it's only done to check an error message, not to check an actual computation. So I am not sure our sample_weight support is correct, even outside of array API concerns.

I guess this is only indirectly tested by classification metrics that rely on multilabel_confusion_matrix internally. But then the array API compliance tests for F1 score do not fail with floating point weights (I just checked) and I am not sure why.

Here is the output of my cuda run on this PR (updated to check that boolean array indexing also works, but this should be orthogonal):

$ pytest -vlx -k "array_api and f1_score" sklearn/ ================================================================================================== test session starts =================================================================================================== platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0 collected 34881 items / 34863 deselected / 2 skipped / 18 selected sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-numpy-None-None] PASSED [ 5%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-array_api_strict-None-None] PASSED [ 11%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-cupy-None-None] PASSED [ 16%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-cupy.array_api-None-None] PASSED [ 22%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cpu-float64] PASSED [ 27%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cpu-float32] PASSED [ 33%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cuda-float64] PASSED [ 38%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-cuda-float32] PASSED [ 44%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_binary_classification_metric-torch-mps-float32] SKIPPED (Skipping MPS device test because PYTORCH_ENABLE_MPS_FALLBACK...) [ 50%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-numpy-None-None] PASSED [ 55%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-array_api_strict-None-None] PASSED [ 61%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-cupy-None-None] PASSED [ 66%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-cupy.array_api-None-None] PASSED [ 72%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cpu-float64] PASSED [ 77%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cpu-float32] PASSED [ 83%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cuda-float64] PASSED [ 88%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-cuda-float32] PASSED [ 94%] sklearn/metrics/tests/test_common.py::test_array_api_compliance[f1_score-check_array_api_multiclass_classification_metric-torch-mps-float32] SKIPPED (Skipping MPS device test because PYTORCH_ENABLE_MPS_FALL...) [100%] ============================================================================= 16 passed, 4 skipped, 34863 deselected, 105 warnings in 15.59s =============================================================================

sklearn/metrics/_classification.py

OmarManzoor · 2024-05-29T09:52:08Z

@ogrisel @betatim Does this look okay now?

ogrisel

I do not have time to finish the review today but here is some quick feedback:

sklearn/metrics/_classification.py

sklearn/utils/_array_api.py

ogrisel · 2024-06-05T15:47:57Z

I merged main to be able to launch the new CUDA GPU CI workflow on this PR:

https://github.com/scikit-learn/scikit-learn/actions/runs/9387261905

EDIT: tests are green.

OmarManzoor · 2024-11-07T09:24:12Z

@adrinjalali Could you kindly have a look at this PR now?

sklearn/metrics/_classification.py

adrinjalali · 2024-11-07T12:40:39Z

sklearn/metrics/_classification.py

+        if _is_numpy_namespace(xp=xp):
+            true_and_pred = y_true.multiply(y_pred)
+        else:
+            true_and_pred = xp.multiply(y_true, y_pred)


why the difference?

The first branch is the case where we are multiplying two sparse matrices together.

then we should check for sparse, not numpy namespace. This looks to me that we'd be using that branch for np.ndarray

sklearn/metrics/_classification.py

sklearn/utils/extmath.py

sklearn/utils/validation.py

OmarManzoor · 2024-11-12T12:05:07Z

@adrinjalali Does this look okay now?

adrinjalali

otherwise LGTM.

sklearn/metrics/_classification.py

adrinjalali · 2024-11-15T10:26:58Z

sklearn/metrics/_classification.py

-        precision = _nanaverage(precision, weights=weights)
-        recall = _nanaverage(recall, weights=weights)
-        f_score = _nanaverage(f_score, weights=weights)
+        assert average != "binary" or precision.shape[0] == 1


I'm okay to leave this as is in this PR with this since it's existing code, but we really shouldn't be assert ing here. If this never happens, then the line shouldn't be here, if it can happen, we should raise a meaningful error.

adrinjalali

Other than the version question, LGTM.

adrinjalali · 2024-11-15T16:27:55Z

sklearn/utils/validation.py

+        `device` object.
+        See the :ref:`Array API User Guide <array_api>` for more details.
+
+        .. versionadded:: 1.6


@glemaitre @jeremiedbb should we change this or backport?

Let's backport since we have the experimental guardrail

glemaitre · 2024-11-25T09:23:36Z

Thanks @OmarManzoor

…cikit-learn#27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

…27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

…cikit-learn#27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

ENH Array API support for f1_score

9bcbc4c

github-actions bot added module:metrics module:preprocessing module:utils labels Sep 14, 2023

OmarManzoor commented Sep 14, 2023

View reviewed changes

sklearn/preprocessing/_label.py Outdated Show resolved Hide resolved

OmarManzoor commented Sep 14, 2023

View reviewed changes

sklearn/utils/_array_api.py Outdated Show resolved Hide resolved

OmarManzoor mentioned this pull request Sep 15, 2023

ENH Array API support for LabelEncoder #27381

Merged

betatim mentioned this pull request Oct 24, 2023

Make more of the "tools" of scikit-learn Array API compatible #26024

Open

OmarManzoor added 5 commits May 17, 2024 12:02

Merge branch 'main' into f1_array_api

87b65b9

Merge branch 'main' into f1_array_api

17457b9

Add array api support for f1_score

c80617f

Add changelog

649ce17

Merge branch 'main' into f1_array_api

8f0db56

OmarManzoor marked this pull request as ready for review May 20, 2024 10:07

OmarManzoor added 3 commits May 20, 2024 16:35

Fix sample weights in _bincount

6a02fcd

Add some fixes

c150c9c

Correct and add tests for nanmean

a01f2d7

OmarManzoor commented May 20, 2024

View reviewed changes

sklearn/metrics/_classification.py Show resolved Hide resolved

OmarManzoor added 2 commits May 20, 2024 18:32

Add options for testing with various average values

aa2f521

Use reshape when creating arrays in micro average

75c7d5a

ogrisel reviewed May 24, 2024

View reviewed changes

OmarManzoor added 2 commits May 27, 2024 09:57

Add LabelEncoder and f1_score in array_api.rst

8b21b51

Merge branch 'main' into f1_array_api

bc8c2df

ogrisel added the Array API label Jun 5, 2024

ogrisel reviewed Jun 5, 2024

View reviewed changes

sklearn/metrics/_classification.py Outdated Show resolved Hide resolved

sklearn/utils/_array_api.py Outdated Show resolved Hide resolved

Merge branch 'main' into f1_array_api

ef33cf6

OmarManzoor added 8 commits October 24, 2024 21:03

Merge branch 'main' into f1_array_api

c828e4d

Add new changelog

a21a875

Add docstring

0c80235

Add xp after array param in _bincount

7b8fb15

Merge branch 'main' into f1_array_api

82cf2bc

Refactor based on PR suggestions

24d91a4

Handle the _tolist method and add doc for device param

c71c974

Use _convert_to_numpy is _tolist

48b24ce

OmarManzoor added the CUDA CI label Nov 7, 2024

github-actions bot removed the CUDA CI label Nov 7, 2024

adrinjalali reviewed Nov 7, 2024

View reviewed changes

OmarManzoor added 3 commits November 7, 2024 18:17

Further PR suggestions

6679ae3

Add sparse check specifically when multiplying

fad41ad

Merge branch 'main' into f1_array_api

8c6c4a0

adrinjalali reviewed Nov 15, 2024

View reviewed changes

OmarManzoor added 2 commits November 15, 2024 18:38

Merge branch 'main' into f1_array_api

0912004

Update 'and' to 'or' condition

88a48e0

adrinjalali approved these changes Nov 15, 2024

View reviewed changes

glemaitre merged commit 96b53ad into scikit-learn:main Nov 25, 2024
30 checks passed

glemaitre added the To backport PR merged in master that need a backport to a release branch defined based on the milestone. label Nov 25, 2024

glemaitre added this to the 1.6 milestone Nov 25, 2024

OmarManzoor deleted the f1_array_api branch November 25, 2024 09:50

OmarManzoor mentioned this pull request Nov 26, 2024

DOC Include precision_recall_fscore_support in array_api #30348

Merged

jeremiedbb pushed a commit to jeremiedbb/scikit-learn that referenced this pull request Dec 4, 2024

ENH Array API support for f1_score and multilabel_confusion_matrix (s…

f0f6be2

…cikit-learn#27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

jeremiedbb pushed a commit that referenced this pull request Dec 6, 2024

ENH Array API support for f1_score and multilabel_confusion_matrix (#…

16ed528

…27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

virchan pushed a commit to virchan/scikit-learn that referenced this pull request Dec 9, 2024

ENH Array API support for f1_score and multilabel_confusion_matrix (s…

36b9315

…cikit-learn#27369) Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

saattrupdan mentioned this pull request Dec 14, 2024

[BUG] return {"f1": float(score) if score.size == 1 else score} EuroEval/EuroEval#643

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH Array API support for f1_score and multilabel_confusion_matrix #27369

ENH Array API support for f1_score and multilabel_confusion_matrix #27369

OmarManzoor commented Sep 14, 2023 •

edited

Loading

github-actions bot commented Sep 14, 2023 •

edited

Loading

OmarManzoor commented May 20, 2024

ogrisel left a comment •

edited

Loading

ogrisel May 24, 2024

ogrisel May 24, 2024

OmarManzoor commented May 29, 2024

ogrisel left a comment

ogrisel commented Jun 5, 2024 •

edited

Loading

OmarManzoor commented Nov 7, 2024

adrinjalali Nov 7, 2024

OmarManzoor Nov 7, 2024

adrinjalali Nov 7, 2024

OmarManzoor Nov 7, 2024

OmarManzoor commented Nov 12, 2024 •

edited

Loading

adrinjalali left a comment

adrinjalali Nov 15, 2024

adrinjalali left a comment

adrinjalali Nov 15, 2024

glemaitre Nov 25, 2024

glemaitre commented Nov 25, 2024

ENH Array API support for f1_score and multilabel_confusion_matrix #27369

ENH Array API support for f1_score and multilabel_confusion_matrix #27369

Conversation

OmarManzoor commented Sep 14, 2023 • edited Loading

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

github-actions bot commented Sep 14, 2023 • edited Loading

✔️ Linting Passed

OmarManzoor commented May 20, 2024

ogrisel left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OmarManzoor commented May 29, 2024

ogrisel left a comment

Choose a reason for hiding this comment

ogrisel commented Jun 5, 2024 • edited Loading

OmarManzoor commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OmarManzoor commented Nov 12, 2024 • edited Loading

adrinjalali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrinjalali left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

glemaitre commented Nov 25, 2024

OmarManzoor commented Sep 14, 2023 •

edited

Loading

github-actions bot commented Sep 14, 2023 •

edited

Loading

ogrisel left a comment •

edited

Loading

ogrisel commented Jun 5, 2024 •

edited

Loading

OmarManzoor commented Nov 12, 2024 •

edited

Loading