Skip to content

Clarification of output array type when metrics accept multiclass/multioutput #31286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lucyleeow opened this issue May 1, 2025 · 0 comments
Labels
Array API Needs Triage Issue requires triage

Comments

@lucyleeow
Copy link
Member

lucyleeow commented May 1, 2025

Clarification of how we should handle array output type when a metric outputs several values (i.e. accepts multiclass or multioutput input).

The issue was summarised succinctly in #30439 (comment):

Not sure what should be the output namespace / device in case we output an array, e.g. roc_auc_score with average=None on multiclass problems...

Currently all regression/classification metrics that support array API and multiclass or multioutput, all output an array in the same namespace and device as the input (checked code and manually). Summary of these metrics :

Regression metrics

Returns array in same namespace/device:

Classification metrics

Returns array in same namespace/device:

Looking at the metrics code, if we wanted to support a list of scalars, we'd generally have to do extra processing to convert an array (often output of an xp.** function) to a list of scalars.

Once we arrive at a consensus we should update the array API documentation and update the check_array_api_metric in tests such that when the output is array/list - we check that the output type etc is correct.

cc @ogrisel @betatim

@github-actions github-actions bot added the Needs Triage Issue requires triage label May 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Array API Needs Triage Issue requires triage
Projects
None yet
Development

No branches or pull requests

1 participant