@Micky774 Could you ping me once you addressed @jjerphan comments such that I make a last review before the merge?

@glemaitre Should be ready for you now.

There are several stylistic considerations that need to be made before merging into main, such as cleaning up Cython code, but let's reserve those for the follow-up PR(s). I think for this one we just need to limit attention to public API. The only potentially controversial aspect of which, right now, is the use of negative labels to distinguish types of outliers.

Currently we do the following:

1. Natively accept np.nan as a valid value
2. Label points -1 if they are noisy outliers
3. Label points -1 if they are infinite outliers, indicated by the presence of +/- np.inf
4. Label points -2 if they are disconnected/missing, indicated by the presence of np.nan

While np.nan usually means missing data, @lmcinnes taught me that there is a meaningful use-case of treating np.nan as an indicator of manifold non-membership (e.g. from UMAP output). See quote from our emails:

Hi Meekail,

The motivation for it was the fact that UMAP added a feature to specify points that are completely disconnected as np.nan values. This is helpful as this integrates with all the plotting libraries people use. The catch was that people like to feed UMAP results directly in hdbscan, and that was resulting in errors which was breaking people's pipelines. From the point of view of what UMAP outputs the np.nan values are simply ultimate outliers, like np.inf really. I do see that data from other sources may present differently, and np.inf doesn't play as nicely with plotting libraries.

Given all of that my opinion on handling would be:

• strictly disallowing nan values is bad, because it will break a lot of common workflows.
• differentiating between nan and inf makes a lot of sense, but wasn't something we had given any thought to.
• Having different classes of noise starts to make a lot of sense really, so I like your -1 and -2 label proposal.
• Could we go even further and use -1 for regular noise, -2 for inf (because they are different) and -3 for nan? I'm not sure how scikit-learn maintainers will feel about that, but it makes a lot of sense to me actually.

Leland.

With this insight in mind, I'm in favor of:

1. Natively accepting np.nan as a valid value (we already do)
2. Label points -1 if they are noisy outliers
3. Label points -2 if they are infinite outliers, indicated by the presence of +/- np.inf
4. Label points -3 if they are disconnected/missing, indicated by the presence of np.nan

For users that care about the semantics distinguishing these, providing them as separate labels would make things smoother to feed into downstream processes. For those that do not care to distinguish them, a simple labels < 0 check suffices.

Curious what everyone's opinions are.

CC: @thomasjpfan

With this insight in mind, I'm in favor of:

1. Natively accepting `np.nan` as a valid value (we already do)

2. Label points `-1` if they are noisy outliers

3. Label points `-2` if they are infinite outliers, indicated by the presence of `+/- np.inf`

4. Label points `-3` if they are disconnected/missing, indicated by the presence of `np.nan`

For users that care about the semantics distinguishing these, providing them as separate labels would make things smoother to feed into downstream processes. For those that do not care to distinguish them, a simple labels < 0 check suffices.

I am OK with this as long as there are constants encoding labels of this labeling and a comment motivating this choice.

For users that care about the semantics distinguishing these, providing them as separate labels would make things smoother to feed into downstream processes. For those that do not care to distinguish them, a simple labels < 0 check suffices.

I am OK with having users be able to use this labelling if it is documented.