-Scikit-learn is helping to drive Moore’s Law, via Solido. Solido creates
+Scikit-learn is helping to drive Moore's Law, via Solido. Solido creates
computer-aided design tools used by the majority of top-20 semiconductor
companies and fabs, to design the bleeding-edge chips inside smartphones,
-automobiles, and more. Scikit-learn helps to power Solido’s algorithms for
+automobiles, and more. Scikit-learn helps to power Solido's algorithms for
rare-event estimation, worst-case verification, optimization, and more. At
-Solido, we are particularly fond of scikit-learn’s libraries for Gaussian
+Solido, we are particularly fond of scikit-learn's libraries for Gaussian
Process models, large-scale regularized linear regression, and classification.
Scikit-learn has increased our productivity, because for many ML problems we no
longer need to “roll our own” code. `This PyData 2014 talk
`_ has details.
diff --git a/doc/tutorial/machine_learning_map/parse_path.py b/doc/tutorial/machine_learning_map/parse_path.py
index 770fd1481f53b..b1c68cec7f76b 100644
--- a/doc/tutorial/machine_learning_map/parse_path.py
+++ b/doc/tutorial/machine_learning_map/parse_path.py
@@ -89,7 +89,7 @@ def convertToFloat(s, loc, toks):
coordinateSequence = Sequence(coordinate)
-coordinatePair = (coordinate + maybeComma + coordinate).setParseAction(lambda t: tuple(t))
+coordinatePair = (coordinate + maybeComma + coordinate).setParseAction(tuple)
coordinatePairSequence = Sequence(coordinatePair)
coordinatePairPair = coordinatePair + maybeComma + coordinatePair
@@ -111,9 +111,9 @@ def convertToFloat(s, loc, toks):
arcRadius = (
nonnegativeNumber + maybeComma + #rx
nonnegativeNumber #ry
-).setParseAction(lambda t: tuple(t))
+).setParseAction(tuple)
-arcFlags = (flag + maybeComma + flag).setParseAction(lambda t: tuple(t))
+arcFlags = (flag + maybeComma + flag).setParseAction(tuple)
ellipticalArcArgument = Group(
arcRadius + maybeComma + #rx, ry
diff --git a/doc/tutorial/statistical_inference/model_selection.rst b/doc/tutorial/statistical_inference/model_selection.rst
index 070e86c18e8b1..dd0cec4de4db0 100644
--- a/doc/tutorial/statistical_inference/model_selection.rst
+++ b/doc/tutorial/statistical_inference/model_selection.rst
@@ -182,7 +182,7 @@ scoring method.
.. topic:: **Exercise**
On the digits dataset, plot the cross-validation score of a :class:`SVC`
- estimator with an linear kernel as a function of parameter ``C`` (use a
+ estimator with a linear kernel as a function of parameter ``C`` (use a
logarithmic grid of points, from 1 to 10).
.. literalinclude:: ../../auto_examples/exercises/plot_cv_digits.py
diff --git a/doc/visualizations.rst b/doc/visualizations.rst
index 0f0ec73549355..0c6590335232a 100644
--- a/doc/visualizations.rst
+++ b/doc/visualizations.rst
@@ -13,7 +13,7 @@ Visualizations
Scikit-learn defines a simple API for creating visualizations for machine
learning. The key feature of this API is to allow for quick plotting and
visual adjustments without recalculation. We provide `Display` classes that
-exposes two methods allowing to make the plotting: `from_estimator` and
+expose two methods for creating plots: `from_estimator` and
`from_predictions`. The `from_estimator` method will take a fitted estimator
and some data (`X` and `y`) and create a `Display` object. Sometimes, we would
like to only compute the predictions once and one should use `from_predictions`
@@ -42,7 +42,7 @@ ROC curve for SVC in future plots. In this case, the `svc_disp` is a
:class:`~sklearn.metrics.RocCurveDisplay` that stores the computed values as
attributes called `roc_auc`, `fpr`, and `tpr`. Be aware that we could get
the predictions from the support vector machine and then use `from_predictions`
-instead of `from_estimator` Next, we train a random forest classifier and plot
+instead of `from_estimator`. Next, we train a random forest classifier and plot
the previously computed roc curve again by using the `plot` method of the
`Display` object.
diff --git a/doc/whats_new.rst b/doc/whats_new.rst
index 931420a30b02f..3354a6b13f32b 100644
--- a/doc/whats_new.rst
+++ b/doc/whats_new.rst
@@ -12,6 +12,7 @@ on libraries.io to be notified when new versions are released.
.. toctree::
:maxdepth: 1
+ Version 1.2
Version 1.1
Version 1.0
Version 0.24
diff --git a/doc/whats_new/older_versions.rst b/doc/whats_new/older_versions.rst
index 63b5c6e9ea4cb..221de4cdb7e4c 100644
--- a/doc/whats_new/older_versions.rst
+++ b/doc/whats_new/older_versions.rst
@@ -1134,7 +1134,7 @@ Changelog
example_gaussian_process_plot_gp_probabilistic_classification_after_regression.py
for a taste of what can be done.
-- It is now possible to use liblinear’s Multi-class SVC (option
+- It is now possible to use liblinear's Multi-class SVC (option
multi_class in :class:`~svm.LinearSVC`)
- New features and performance improvements of text feature
diff --git a/doc/whats_new/v1.1.rst b/doc/whats_new/v1.1.rst
index 0ce67e9d1f8d5..27fa545901dd0 100644
--- a/doc/whats_new/v1.1.rst
+++ b/doc/whats_new/v1.1.rst
@@ -2,6 +2,106 @@
.. currentmodule:: sklearn
+.. _changes_1_1_2:
+
+Version 1.1.2
+=============
+
+**August 2022**
+
+Changed models
+--------------
+
+The following estimators and functions, when fit with the same data and
+parameters, may produce different models from the previous version. This often
+occurs due to changes in the modelling logic (bug fixes or enhancements), or in
+random sampling procedures.
+
+- |Fix| :class:`manifold.TSNE` now throws a `ValueError` when fit with
+ `perplexity>=n_samples` to ensure mathematical correctness of the algorithm.
+ :pr:`10805` by :user:`Mathias Andersen ` and
+ :pr:`23471` by :user:`Meekail Zain `.
+
+Changelog
+---------
+
+- |Fix| A default HTML representation is shown for meta-estimators with invalid
+ parameters. :pr:`24015` by `Thomas Fan`_.
+
+- |Fix| Add support for F-contiguous arrays for estimators and functions whose back-end
+ have been changed in 1.1.
+ :pr:`23990` by :user:`Julien Jerphanion `.
+
+- |Fix| Wheels are now available for MacOS 10.9 and greater. :pr:`23833` by
+ `Thomas Fan`_.
+
+:mod:`sklearn.base`
+...................
+
+- |Fix| The `get_params` method of the :class:`BaseEstimator` class now supports
+ estimators with `type`-type params that have the `get_params` method.
+ :pr:`24017` by :user:`Henry Sorsky `.
+
+:mod:`sklearn.cluster`
+......................
+
+- |Fix| Fixed a bug in :class:`cluster.Birch` that could trigger an error when splitting
+ a node if there are duplicates in the dataset.
+ :pr:`23395` by :user:`Jérémie du Boisberranger `.
+
+:mod:`sklearn.feature_selection`
+................................
+
+- |Fix| :class:`feature_selection.SelectFromModel` defaults to selection
+ threshold 1e-5 when the estimator is either :class:`linear_model.ElasticNet`
+ or :class:`linear_model.ElasticNetCV` with `l1_ratio` equals 1 or
+ :class:`linear_model.LassoCV`.
+ :pr:`23636` by :user:`Hao Chun Chang `.
+
+:mod:`sklearn.impute`
+.....................
+
+- |Fix| :class:`impute.SimpleImputer` uses the dtype seen in `fit` for
+ `transform` when the dtype is object. :pr:`22063` by `Thomas Fan`_.
+
+:mod:`sklearn.linear_model`
+...........................
+
+- |Fix| Use dtype-aware tolerances for the validation of gram matrices (passed by users
+ or precomputed). :pr:`22059` by :user:`Malte S. Kurz `.
+
+- |Fix| Fixed an error in :class:`linear_model.LogisticRegression` with
+ `solver="newton-cg"`, `fit_intercept=True`, and a single feature. :pr:`23608`
+ by `Tom Dupre la Tour`_.
+
+:mod:`sklearn.manifold`
+.......................
+
+- |Fix| :class:`manifold.TSNE` now throws a `ValueError` when fit with
+ `perplexity>=n_samples` to ensure mathematical correctness of the algorithm.
+ :pr:`10805` by :user:`Mathias Andersen ` and
+ :pr:`23471` by :user:`Meekail Zain `.
+
+:mod:`sklearn.metrics`
+......................
+
+- |Fix| Fixed error message of :class:`metrics.coverage_error` for 1D array input.
+ :pr:`23548` by :user:`Hao Chun Chang `.
+
+:mod:`sklearn.preprocessing`
+............................
+
+- |Fix| :meth:`preprocessing.OrdinalEncoder.inverse_transform` correctly handles
+ use cases where `unknown_value` or `encoded_missing_value` is `nan`. :pr:`24087`
+ by `Thomas Fan`_.
+
+:mod:`sklearn.tree`
+...................
+
+- |Fix| Fixed invalid memory access bug during fit in
+ :class:`tree.DecisionTreeRegressor` and :class:`tree.DecisionTreeClassifier`.
+ :pr:`23273` by `Thomas Fan`_.
+
.. _changes_1_1_1:
Version 1.1.1
@@ -38,6 +138,19 @@ Changelog
:mod:`sklearn.feature_selection`
................................
+- |Fix| The `partial_fit` method of :class:`feature_selection.SelectFromModel`
+ now conducts validation for `max_features` and `feature_names_in` parameters.
+ :pr:`23299` by :user:`Long Bao `.
+
+:mod:`sklearn.decomposition`
+............................
+
+- |Fix| Avoid spurious warning in :class:`decomposition.IncrementalPCA` when
+ `n_samples == n_components`. :pr:`23264` by :user:`Lucy Liu `.
+
+:mod:`sklearn.feature_selection`
+................................
+
- |Fix| The `partial_fit` method of :class:`feature_selection.SelectFromModel`
now conducts validation for `max_features` and `feature_names_in` parameters.
:pr:`23299` by :user:`Long Bao `.
@@ -609,6 +722,9 @@ Changelog
:class:`ensemble.ExtraTreesClassifier`.
:pr:`20803` by :user:`Brian Sun `.
+- |Efficiency| Improve runtime performance of :class:`ensemble.IsolationForest`
+ by skipping repetitive input checks. :pr:`23149` by :user:`Zhehao Liu `.
+
:mod:`sklearn.feature_extraction`
.................................
diff --git a/examples/applications/plot_model_complexity_influence.py b/examples/applications/plot_model_complexity_influence.py
index d05f4ab497ada..60475e2c4302e 100644
--- a/examples/applications/plot_model_complexity_influence.py
+++ b/examples/applications/plot_model_complexity_influence.py
@@ -21,8 +21,12 @@
- :class:`~sklearn.svm.NuSVR` (for regression data) which implements
Nu support vector regression;
- - :class:`~sklearn.ensemble.GradientBoostingRegressor` (for regression
- data) which builds an additive model in a forward stage-wise fashion.
+ - :class:`~sklearn.ensemble.GradientBoostingRegressor` builds an additive
+ model in a forward stage-wise fashion. Notice that
+ :class:`~sklearn.ensemble.HistGradientBoostingRegressor` is much faster
+ than :class:`~sklearn.ensemble.GradientBoostingRegressor` starting with
+ intermediate datasets (`n_samples >= 10_000`), which is not the case for
+ this example.
We make the model complexity vary through the choice of relevant model
diff --git a/examples/classification/plot_lda.py b/examples/classification/plot_lda.py
index 47487fc1f2caf..4213fc614a31a 100644
--- a/examples/classification/plot_lda.py
+++ b/examples/classification/plot_lda.py
@@ -71,6 +71,7 @@ def generate_data(n_samples, n_features):
linewidth=2,
label="Linear Discriminant Analysis with Ledoit Wolf",
color="navy",
+ linestyle="dashed",
)
plt.plot(
features_samples_ratio,
@@ -78,6 +79,7 @@ def generate_data(n_samples, n_features):
linewidth=2,
label="Linear Discriminant Analysis",
color="gold",
+ linestyle="solid",
)
plt.plot(
features_samples_ratio,
@@ -85,6 +87,7 @@ def generate_data(n_samples, n_features):
linewidth=2,
label="Linear Discriminant Analysis with OAS",
color="red",
+ linestyle="dotted",
)
plt.xlabel("n_features / n_samples")
diff --git a/examples/cluster/plot_agglomerative_clustering.py b/examples/cluster/plot_agglomerative_clustering.py
index 9d590f572f121..5bb87a9386bf8 100644
--- a/examples/cluster/plot_agglomerative_clustering.py
+++ b/examples/cluster/plot_agglomerative_clustering.py
@@ -6,7 +6,7 @@
local structure in the data. The graph is simply the graph of 20 nearest
neighbors.
-Two consequences of imposing a connectivity can be seen. First, clustering
+There are two advantages of imposing a connectivity. First, clustering
without a connectivity matrix is much faster.
Second, when using a connectivity matrix, single, average and complete
diff --git a/examples/compose/plot_digits_pipe.py b/examples/compose/plot_digits_pipe.py
index acd3068d991c9..dcedfe0da2beb 100644
--- a/examples/compose/plot_digits_pipe.py
+++ b/examples/compose/plot_digits_pipe.py
@@ -37,7 +37,7 @@
pipe = Pipeline(steps=[("scaler", scaler), ("pca", pca), ("logistic", logistic)])
X_digits, y_digits = datasets.load_digits(return_X_y=True)
-# Parameters of pipelines can be set using ‘__’ separated parameter names:
+# Parameters of pipelines can be set using '__' separated parameter names:
param_grid = {
"pca__n_components": [5, 15, 30, 45, 60],
"logistic__C": np.logspace(-4, 4, 4),
diff --git a/examples/decomposition/plot_faces_decomposition.py b/examples/decomposition/plot_faces_decomposition.py
index 0eb07dc3efb2d..c21ac347c0e06 100644
--- a/examples/decomposition/plot_faces_decomposition.py
+++ b/examples/decomposition/plot_faces_decomposition.py
@@ -79,7 +79,7 @@ def plot_gallery(title, images, n_col=n_col, n_row=n_row, cmap=plt.cm.gray):
# %%
-# Let’s take a look at our data. Gray color indicates negative values,
+# Let's take a look at our data. Gray color indicates negative values,
# white indicates positive values.
plot_gallery("Faces from dataset", faces_centered[:n_components])
diff --git a/examples/ensemble/plot_feature_transformation.py b/examples/ensemble/plot_feature_transformation.py
index 409396a0376b8..36eb87bb757cd 100644
--- a/examples/ensemble/plot_feature_transformation.py
+++ b/examples/ensemble/plot_feature_transformation.py
@@ -39,7 +39,7 @@
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
-X, y = make_classification(n_samples=80000, random_state=10)
+X, y = make_classification(n_samples=80_000, random_state=10)
X_full_train, X_test, y_full_train, y_test = train_test_split(
X, y, test_size=0.5, random_state=10
@@ -72,6 +72,11 @@
_ = gradient_boosting.fit(X_train_ensemble, y_train_ensemble)
# %%
+# Notice that :class:`~sklearn.ensemble.HistGradientBoostingClassifier` is much
+# faster than :class:`~sklearn.ensemble.GradientBoostingClassifier` starting
+# with intermediate datasets (`n_samples >= 10_000`), which is not the case of
+# the present example.
+#
# The :class:`~sklearn.ensemble.RandomTreesEmbedding` is an unsupervised method
# and thus does not required to be trained independently.
diff --git a/examples/ensemble/plot_gradient_boosting_quantile.py b/examples/ensemble/plot_gradient_boosting_quantile.py
index 9e823439b948b..2aa04c3988d9e 100644
--- a/examples/ensemble/plot_gradient_boosting_quantile.py
+++ b/examples/ensemble/plot_gradient_boosting_quantile.py
@@ -72,6 +72,11 @@ def f(x):
all_models["q %1.2f" % alpha] = gbr.fit(X_train, y_train)
# %%
+# Notice that :class:`~sklearn.ensemble.HistGradientBoostingRegressor` is much
+# faster than :class:`~sklearn.ensemble.GradientBoostingRegressor` starting with
+# intermediate datasets (`n_samples >= 10_000`), which is not the case of the
+# present example.
+#
# For the sake of comparison, we also fit a baseline model trained with the
# usual (mean) squared error (MSE).
gbr_ls = GradientBoostingRegressor(loss="squared_error", **common_params)
diff --git a/examples/gaussian_process/plot_gpc_xor.py b/examples/gaussian_process/plot_gpc_xor.py
index 6eebbcf80098e..6e6217dba8b9e 100644
--- a/examples/gaussian_process/plot_gpc_xor.py
+++ b/examples/gaussian_process/plot_gpc_xor.py
@@ -29,7 +29,7 @@
# fit the model
plt.figure(figsize=(10, 5))
-kernels = [1.0 * RBF(length_scale=1.0), 1.0 * DotProduct(sigma_0=1.0) ** 2]
+kernels = [1.0 * RBF(length_scale=1.15), 1.0 * DotProduct(sigma_0=1.0) ** 2]
for i, kernel in enumerate(kernels):
clf = GaussianProcessClassifier(kernel=kernel, warm_start=True).fit(X, Y)
diff --git a/examples/gaussian_process/plot_gpr_noisy.py b/examples/gaussian_process/plot_gpr_noisy.py
index 04ea696e4319f..e15c9a6470d38 100644
--- a/examples/gaussian_process/plot_gpr_noisy.py
+++ b/examples/gaussian_process/plot_gpr_noisy.py
@@ -97,7 +97,7 @@ def target_generator(X, add_noise=False):
# %%
plt.plot(X, y, label="Expected signal")
-plt.scatter(x=X_train[:, 0], y=y_train, color="black", alpha=0.4, label="Observsations")
+plt.scatter(x=X_train[:, 0], y=y_train, color="black", alpha=0.4, label="Observations")
plt.errorbar(X, y_mean, y_std)
plt.legend()
plt.xlabel("X")
diff --git a/examples/gaussian_process/plot_gpr_prior_posterior.py b/examples/gaussian_process/plot_gpr_prior_posterior.py
index 437d67f5b0ab9..b36d49617432d 100644
--- a/examples/gaussian_process/plot_gpr_prior_posterior.py
+++ b/examples/gaussian_process/plot_gpr_prior_posterior.py
@@ -158,7 +158,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
)
# %%
-# Periodic kernel
+# Exp-Sine-Squared kernel
# ...............
from sklearn.gaussian_process.kernels import ExpSineSquared
@@ -183,7 +183,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left")
axs[1].set_title("Samples from posterior distribution")
-fig.suptitle("Periodic kernel", fontsize=18)
+fig.suptitle("Exp-Sine-Squared kernel", fontsize=18)
plt.tight_layout()
# %%
@@ -194,7 +194,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
)
# %%
-# Dot product kernel
+# Dot-product kernel
# ..................
from sklearn.gaussian_process.kernels import ConstantKernel, DotProduct
@@ -216,7 +216,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left")
axs[1].set_title("Samples from posterior distribution")
-fig.suptitle("Dot product kernel", fontsize=18)
+fig.suptitle("Dot-product kernel", fontsize=18)
plt.tight_layout()
# %%
@@ -227,7 +227,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
)
# %%
-# Mattern kernel
+# Matérn kernel
# ..............
from sklearn.gaussian_process.kernels import Matern
@@ -247,7 +247,7 @@ def plot_gpr_samples(gpr_model, n_samples, ax):
axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left")
axs[1].set_title("Samples from posterior distribution")
-fig.suptitle("Mattern kernel", fontsize=18)
+fig.suptitle("Matérn kernel", fontsize=18)
plt.tight_layout()
# %%
diff --git a/examples/inspection/plot_permutation_importance.py b/examples/inspection/plot_permutation_importance.py
index 53e4969f1740e..cd6823adf5f9b 100644
--- a/examples/inspection/plot_permutation_importance.py
+++ b/examples/inspection/plot_permutation_importance.py
@@ -58,7 +58,7 @@
# We define a predictive model based on a random forest. Therefore, we will make
# the following preprocessing steps:
#
-# - use :class:`~sklearn.preprocessing.OrdinaleEcnoder` to encode the
+# - use :class:`~sklearn.preprocessing.OrdinalEncoder` to encode the
# categorical features;
# - use :class:`~sklearn.impute.SimpleImputer` to fill missing values for
# numerical features using a mean strategy.
diff --git a/examples/model_selection/plot_grid_search_digits.py b/examples/model_selection/plot_grid_search_digits.py
index 2aaa64043749b..ec4360692aaf3 100644
--- a/examples/model_selection/plot_grid_search_digits.py
+++ b/examples/model_selection/plot_grid_search_digits.py
@@ -1,6 +1,6 @@
"""
============================================================
-Parameter estimation using grid search with cross-validation
+Custom refit strategy of a grid search with cross-validation
============================================================
This examples shows how a classifier is optimized by cross-validation,
@@ -13,56 +13,196 @@
More details on tools available for model selection can be found in the
sections on :ref:`cross_validation` and :ref:`grid_search`.
-
"""
+# %%
+# The dataset
+# -----------
+#
+# We will work with the `digits` dataset. The goal is to classify handwritten
+# digits images.
+# We transform the problem into a binary classification for easier
+# understanding: the goal is to identify whether a digit is `8` or not.
from sklearn import datasets
-from sklearn.model_selection import train_test_split
-from sklearn.model_selection import GridSearchCV
-from sklearn.metrics import classification_report
-from sklearn.svm import SVC
-# Loading the Digits dataset
-X, y = datasets.load_digits(return_X_y=True)
+digits = datasets.load_digits()
+
+# %%
+# In order to train a classifier on images, we need to flatten them into vectors.
+# Each image of 8 by 8 pixels needs to be transformed to a vector of 64 pixels.
+# Thus, we will get a final data array of shape `(n_images, n_pixels)`.
+n_samples = len(digits.images)
+X = digits.images.reshape((n_samples, -1))
+y = digits.target == 8
+print(
+ f"The number of images is {X.shape[0]} and each image contains {X.shape[1]} pixels"
+)
+
+# %%
+# As presented in the introduction, the data will be split into a training
+# and a testing set of equal size.
+from sklearn.model_selection import train_test_split
-# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)
-# Set the parameters by cross-validation
+# %%
+# Define our grid-search strategy
+# -------------------------------
+#
+# We will select a classifier by searching the best hyper-parameters on folds
+# of the training set. To do this, we need to define
+# the scores to select the best candidate.
+
+scores = ["precision", "recall"]
+
+# %%
+# We can also define a function to be passed to the `refit` parameter of the
+# :class:`~sklearn.model_selection.GridSearchCV` instance. It will implement the
+# custom strategy to select the best candidate from the `cv_results_` attribute
+# of the :class:`~sklearn.model_selection.GridSearchCV`. Once the candidate is
+# selected, it is automatically refitted by the
+# :class:`~sklearn.model_selection.GridSearchCV` instance.
+#
+# Here, the strategy is to short-list the models which are the best in terms of
+# precision and recall. From the selected models, we finally select the fastest
+# model at predicting. Notice that these custom choices are completely
+# arbitrary.
+
+import pandas as pd
+
+
+def print_dataframe(filtered_cv_results):
+ """Pretty print for filtered dataframe"""
+ for mean_precision, std_precision, mean_recall, std_recall, params in zip(
+ filtered_cv_results["mean_test_precision"],
+ filtered_cv_results["std_test_precision"],
+ filtered_cv_results["mean_test_recall"],
+ filtered_cv_results["std_test_recall"],
+ filtered_cv_results["params"],
+ ):
+ print(
+ f"precision: {mean_precision:0.3f} (±{std_precision:0.03f}),"
+ f" recall: {mean_recall:0.3f} (±{std_recall:0.03f}),"
+ f" for {params}"
+ )
+ print()
+
+
+def refit_strategy(cv_results):
+ """Define the strategy to select the best estimator.
+
+ The strategy defined here is to filter-out all results below a precision threshold
+ of 0.98, rank the remaining by recall and keep all models with one standard
+ deviation of the best by recall. Once these models are selected, we can select the
+ fastest model to predict.
+
+ Parameters
+ ----------
+ cv_results : dict of numpy (masked) ndarrays
+ CV results as returned by the `GridSearchCV`.
+
+ Returns
+ -------
+ best_index : int
+ The index of the best estimator as it appears in `cv_results`.
+ """
+ # print the info about the grid-search for the different scores
+ precision_threshold = 0.98
+
+ cv_results_ = pd.DataFrame(cv_results)
+ print("All grid-search results:")
+ print_dataframe(cv_results_)
+
+ # Filter-out all results below the threshold
+ high_precision_cv_results = cv_results_[
+ cv_results_["mean_test_precision"] > precision_threshold
+ ]
+
+ print(f"Models with a precision higher than {precision_threshold}:")
+ print_dataframe(high_precision_cv_results)
+
+ high_precision_cv_results = high_precision_cv_results[
+ [
+ "mean_score_time",
+ "mean_test_recall",
+ "std_test_recall",
+ "mean_test_precision",
+ "std_test_precision",
+ "rank_test_recall",
+ "rank_test_precision",
+ "params",
+ ]
+ ]
+
+ # Select the most performant models in terms of recall
+ # (within 1 sigma from the best)
+ best_recall_std = high_precision_cv_results["mean_test_recall"].std()
+ best_recall = high_precision_cv_results["mean_test_recall"].max()
+ best_recall_threshold = best_recall - best_recall_std
+
+ high_recall_cv_results = high_precision_cv_results[
+ high_precision_cv_results["mean_test_recall"] > best_recall_threshold
+ ]
+ print(
+ "Out of the previously selected high precision models, we keep all the\n"
+ "the models within one standard deviation of the highest recall model:"
+ )
+ print_dataframe(high_recall_cv_results)
+
+ # From the best candidates, select the fastest model to predict
+ fastest_top_recall_high_precision_index = high_recall_cv_results[
+ "mean_score_time"
+ ].idxmin()
+
+ print(
+ "\nThe selected final model is the fastest to predict out of the previously\n"
+ "selected subset of best models based on precision and recall.\n"
+ "Its scoring time is:\n\n"
+ f"{high_recall_cv_results.loc[fastest_top_recall_high_precision_index]}"
+ )
+
+ return fastest_top_recall_high_precision_index
+
+
+# %%
+#
+# Tuning hyper-parameters
+# -----------------------
+#
+# Once we defined our strategy to select the best model, we define the values
+# of the hyper-parameters and create the grid-search instance:
+from sklearn.model_selection import GridSearchCV
+from sklearn.svm import SVC
+
tuned_parameters = [
{"kernel": ["rbf"], "gamma": [1e-3, 1e-4], "C": [1, 10, 100, 1000]},
{"kernel": ["linear"], "C": [1, 10, 100, 1000]},
]
-scores = ["precision", "recall"]
-
-for score in scores:
- print("# Tuning hyper-parameters for %s" % score)
- print()
+grid_search = GridSearchCV(
+ SVC(), tuned_parameters, scoring=scores, refit=refit_strategy
+)
+grid_search.fit(X_train, y_train)
- clf = GridSearchCV(SVC(), tuned_parameters, scoring="%s_macro" % score)
- clf.fit(X_train, y_train)
+# %%
+#
+# The parameters selected by the grid-search with our custom strategy are:
+grid_search.best_params_
- print("Best parameters set found on development set:")
- print()
- print(clf.best_params_)
- print()
- print("Grid scores on development set:")
- print()
- means = clf.cv_results_["mean_test_score"]
- stds = clf.cv_results_["std_test_score"]
- for mean, std, params in zip(means, stds, clf.cv_results_["params"]):
- print("%0.3f (+/-%0.03f) for %r" % (mean, std * 2, params))
- print()
+# %%
+#
+# Finally, we evaluate the fine-tuned model on the left-out evaluation set: the
+# `grid_search` object **has automatically been refit** on the full training
+# set with the parameters selected by our custom refit strategy.
+#
+# We can use the classification report to compute standard classification
+# metrics on the left-out set:
+from sklearn.metrics import classification_report
- print("Detailed classification report:")
- print()
- print("The model is trained on the full development set.")
- print("The scores are computed on the full evaluation set.")
- print()
- y_true, y_pred = y_test, clf.predict(X_test)
- print(classification_report(y_true, y_pred))
- print()
+y_pred = grid_search.predict(X_test)
+print(classification_report(y_test, y_pred))
-# Note the problem is too easy: the hyperparameter plateau is too flat and the
-# output model is the same for precision and recall with ties in quality.
+# %%
+# .. note::
+# The problem is too easy: the hyperparameter plateau is too flat and the
+# output model is the same for precision and recall with ties in quality.
diff --git a/examples/model_selection/plot_learning_curve.py b/examples/model_selection/plot_learning_curve.py
index 25f43d8b8a3e4..5430f673d76a5 100644
--- a/examples/model_selection/plot_learning_curve.py
+++ b/examples/model_selection/plot_learning_curve.py
@@ -35,6 +35,7 @@ def plot_learning_curve(
ylim=None,
cv=None,
n_jobs=None,
+ scoring=None,
train_sizes=np.linspace(0.1, 1.0, 5),
):
"""
@@ -86,6 +87,11 @@ def plot_learning_curve(
``-1`` means using all processors. See :term:`Glossary `
for more details.
+ scoring : str or callable, default=None
+ A str (see model evaluation documentation) or
+ a scorer callable object / function with signature
+ ``scorer(estimator, X, y)``.
+
train_sizes : array-like of shape (n_ticks,)
Relative or absolute numbers of training examples that will be used to
generate the learning curve. If the ``dtype`` is float, it is regarded
@@ -109,6 +115,7 @@ def plot_learning_curve(
estimator,
X,
y,
+ scoring=scoring,
cv=cv,
n_jobs=n_jobs,
train_sizes=train_sizes,
@@ -189,7 +196,15 @@ def plot_learning_curve(
estimator = GaussianNB()
plot_learning_curve(
- estimator, title, X, y, axes=axes[:, 0], ylim=(0.7, 1.01), cv=cv, n_jobs=4
+ estimator,
+ title,
+ X,
+ y,
+ axes=axes[:, 0],
+ ylim=(0.7, 1.01),
+ cv=cv,
+ n_jobs=4,
+ scoring="accuracy",
)
title = r"Learning Curves (SVM, RBF kernel, $\gamma=0.001$)"
diff --git a/examples/neighbors/plot_species_kde.py b/examples/neighbors/plot_species_kde.py
index c409d354ec986..35ea40158a45c 100644
--- a/examples/neighbors/plot_species_kde.py
+++ b/examples/neighbors/plot_species_kde.py
@@ -19,7 +19,7 @@
The two species are:
- `"Bradypus variegatus"
- `_ ,
+ `_ ,
the Brown-throated Sloth.
- `"Microryzomys minutus"
diff --git a/examples/preprocessing/plot_all_scaling.py b/examples/preprocessing/plot_all_scaling.py
index 49af744011d12..d8a20ece5c56c 100644
--- a/examples/preprocessing/plot_all_scaling.py
+++ b/examples/preprocessing/plot_all_scaling.py
@@ -324,7 +324,7 @@ def make_plot(item_idx):
#
# Unlike the previous scalers, the centering and scaling statistics of
# :class:`~sklearn.preprocessing.RobustScaler`
-# is based on percentiles and are therefore not influenced by a few
+# are based on percentiles and are therefore not influenced by a small
# number of very large marginal outliers. Consequently, the resulting range of
# the transformed feature values is larger than for the previous scalers and,
# more importantly, are approximately similar: for both features most of the
diff --git a/examples/text/plot_document_classification_20newsgroups.py b/examples/text/plot_document_classification_20newsgroups.py
index 7f24861a0e9ce..7eb14a94a724f 100644
--- a/examples/text/plot_document_classification_20newsgroups.py
+++ b/examples/text/plot_document_classification_20newsgroups.py
@@ -3,45 +3,42 @@
Classification of text documents using sparse features
======================================================
-This is an example showing how scikit-learn can be used to classify documents
-by topics using a bag-of-words approach. This example uses a scipy.sparse
-matrix to store the features and demonstrates various classifiers that can
-efficiently handle sparse matrices.
+This is an example showing how scikit-learn can be used to classify documents by
+topics using a `Bag of Words approach
+`_. This example uses a
+Tf-idf-weighted document-term sparse matrix to encode the features and
+demonstrates various classifiers that can efficiently handle sparse matrices.
-The dataset used in this example is the 20 newsgroups dataset. It will be
-automatically downloaded, then cached.
+For document analysis via an unsupervised learning approach, see the example
+script :ref:`sphx_glr_auto_examples_text_plot_document_clustering.py`.
"""
# Author: Peter Prettenhofer
# Olivier Grisel
# Mathieu Blondel
+# Arturo Amor
# Lars Buitinck
# License: BSD 3 clause
# %%
-# Configuration options for the analysis
-# --------------------------------------
-
-# If True, we use `HashingVectorizer`, otherwise we use a `TfidfVectorizer`
-USE_HASHING = False
-
-# Number of features used by `HashingVectorizer`
-N_FEATURES = 2**16
-
-# Optional feature selection: either False, or an integer: the number of
-# features to select
-SELECT_CHI2 = False
-
+# Loading and vectorizing the 20 newsgroups text dataset
+# ======================================================
+#
+# We define a function to load data from :ref:`20newsgroups_dataset`, which
+# comprises around 18,000 newsgroups posts on 20 topics split in two subsets:
+# one for training (or development) and the other one for testing (or for
+# performance evaluation). Note that, by default, the text samples contain some
+# message metadata such as `'headers'`, `'footers'` (signatures) and `'quotes'`
+# to other posts. The `fetch_20newsgroups` function therefore accepts a
+# parameter named `remove` to attempt stripping such information that can make
+# the classification problem "too easy". This is achieved using simple
+# heuristics that are neither perfect nor standard, hence disabled by default.
-# %%
-# Load data from the training set
-# ------------------------------------
-# Let's load data from the newsgroups dataset which comprises around 18000
-# newsgroups posts on 20 topics split in two subsets: one for training (or
-# development) and the other one for testing (or for performance evaluation).
from sklearn.datasets import fetch_20newsgroups
+from sklearn.feature_extraction.text import TfidfVectorizer
+from time import time
categories = [
"alt.atheism",
@@ -50,157 +47,322 @@
"sci.space",
]
-data_train = fetch_20newsgroups(
- subset="train", categories=categories, shuffle=True, random_state=42
-)
-data_test = fetch_20newsgroups(
- subset="test", categories=categories, shuffle=True, random_state=42
-)
-print("data loaded")
+def size_mb(docs):
+ return sum(len(s.encode("utf-8")) for s in docs) / 1e6
-# order of labels in `target_names` can be different from `categories`
-target_names = data_train.target_names
+def load_dataset(verbose=False, remove=()):
+ """Load and vectorize the 20 newsgroups dataset."""
-def size_mb(docs):
- return sum(len(s.encode("utf-8")) for s in docs) / 1e6
+ data_train = fetch_20newsgroups(
+ subset="train",
+ categories=categories,
+ shuffle=True,
+ random_state=42,
+ remove=remove,
+ )
+ data_test = fetch_20newsgroups(
+ subset="test",
+ categories=categories,
+ shuffle=True,
+ random_state=42,
+ remove=remove,
+ )
-data_train_size_mb = size_mb(data_train.data)
-data_test_size_mb = size_mb(data_test.data)
+ # order of labels in `target_names` can be different from `categories`
+ target_names = data_train.target_names
+
+ # split target in a training set and a test set
+ y_train, y_test = data_train.target, data_test.target
+
+ # Extracting features from the training data using a sparse vectorizer
+ t0 = time()
+ vectorizer = TfidfVectorizer(
+ sublinear_tf=True, max_df=0.5, min_df=5, stop_words="english"
+ )
+ X_train = vectorizer.fit_transform(data_train.data)
+ duration_train = time() - t0
+
+ # Extracting features from the test data using the same vectorizer
+ t0 = time()
+ X_test = vectorizer.transform(data_test.data)
+ duration_test = time() - t0
+
+ feature_names = vectorizer.get_feature_names_out()
+
+ if verbose:
+
+ # compute size of loaded data
+ data_train_size_mb = size_mb(data_train.data)
+ data_test_size_mb = size_mb(data_test.data)
+
+ print(
+ f"{len(data_train.data)} documents - "
+ f"{data_train_size_mb:.2f}MB (training set)"
+ )
+ print(f"{len(data_test.data)} documents - {data_test_size_mb:.2f}MB (test set)")
+ print(f"{len(target_names)} categories")
+ print(
+ f"vectorize training done in {duration_train:.3f}s "
+ f"at {data_train_size_mb / duration_train:.3f}MB/s"
+ )
+ print(f"n_samples: {X_train.shape[0]}, n_features: {X_train.shape[1]}")
+ print(
+ f"vectorize testing done in {duration_test:.3f}s "
+ f"at {data_test_size_mb / duration_test:.3f}MB/s"
+ )
+ print(f"n_samples: {X_test.shape[0]}, n_features: {X_test.shape[1]}")
+
+ return X_train, X_test, y_train, y_test, feature_names, target_names
-print(
- "%d documents - %0.3fMB (training set)" % (len(data_train.data), data_train_size_mb)
-)
-print("%d documents - %0.3fMB (test set)" % (len(data_test.data), data_test_size_mb))
-print("%d categories" % len(target_names))
# %%
-# Vectorize the training and test data
-# -------------------------------------
+# Analysis of a bag-of-words document classifier
+# ==============================================
+#
+# We will now train a classifier twice, once on the text samples including
+# metadata and once after stripping the metadata. For both cases we will analyze
+# the classification errors on a test set using a confusion matrix and inspect
+# the coefficients that define the classification function of the trained
+# models.
#
-# split a training set and a test set
-y_train, y_test = data_train.target, data_test.target
+# Model without metadata stripping
+# --------------------------------
+#
+# We start by using the custom function `load_dataset` to load the data without
+# metadata stripping.
+
+X_train, X_test, y_train, y_test, feature_names, target_names = load_dataset(
+ verbose=True
+)
# %%
-# Extracting features from the training data using a sparse vectorizer
-from time import time
+# Our first model is an instance of the
+# :class:`~sklearn.linear_model.RidgeClassifier` class. This is a linear
+# classification model that uses the mean squared error on {-1, 1} encoded
+# targets, one for each possible class. Contrary to
+# :class:`~sklearn.linear_model.LogisticRegression`,
+# :class:`~sklearn.linear_model.RidgeClassifier` does not
+# provide probabilistic predictions (no `predict_proba` method),
+# but it is often faster to train.
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.feature_extraction.text import HashingVectorizer
+from sklearn.linear_model import RidgeClassifier
+
+clf = RidgeClassifier(tol=1e-2, solver="sparse_cg")
+clf.fit(X_train, y_train)
+pred = clf.predict(X_test)
+
+# %%
+# We plot the confusion matrix of this classifier to find if there is a pattern
+# in the classification errors.
-t0 = time()
+import matplotlib.pyplot as plt
+from sklearn.metrics import ConfusionMatrixDisplay
+
+fig, ax = plt.subplots(figsize=(10, 5))
+ConfusionMatrixDisplay.from_predictions(y_test, pred, ax=ax)
+ax.xaxis.set_ticklabels(target_names)
+ax.yaxis.set_ticklabels(target_names)
+_ = ax.set_title(
+ f"Confusion Matrix for {clf.__class__.__name__}\non the original documents"
+)
+
+# %%
+# The confusion matrix highlights that documents of the `alt.atheism` class are
+# often confused with documents with the class `talk.religion.misc` class and
+# vice-versa which is expected since the topics are semantically related.
+#
+# We also observe that some documents of the `sci.space` class can be misclassified as
+# `comp.graphics` while the converse is much rarer. A manual inspection of those
+# badly classified documents would be required to get some insights on this
+# asymmetry. It could be the case that the vocabulary of the space topic could
+# be more specific than the vocabulary for computer graphics.
+#
+# We can gain a deeper understanding of how this classifier makes its decisions
+# by looking at the words with the highest average feature effects:
-if USE_HASHING:
- vectorizer = HashingVectorizer(
- stop_words="english", alternate_sign=False, n_features=N_FEATURES
+import pandas as pd
+import numpy as np
+
+
+def plot_feature_effects():
+ # learned coefficients weighted by frequency of appearance
+ average_feature_effects = clf.coef_ * np.asarray(X_train.mean(axis=0)).ravel()
+
+ for i, label in enumerate(target_names):
+ top5 = np.argsort(average_feature_effects[i])[-5:][::-1]
+ if i == 0:
+ top = pd.DataFrame(feature_names[top5], columns=[label])
+ top_indices = top5
+ else:
+ top[label] = feature_names[top5]
+ top_indices = np.concatenate((top_indices, top5), axis=None)
+ top_indices = np.unique(top_indices)
+ predictive_words = feature_names[top_indices]
+
+ # plot feature effects
+ bar_size = 0.25
+ padding = 0.75
+ y_locs = np.arange(len(top_indices)) * (4 * bar_size + padding)
+
+ fig, ax = plt.subplots(figsize=(10, 8))
+ for i, label in enumerate(target_names):
+ ax.barh(
+ y_locs + (i - 2) * bar_size,
+ average_feature_effects[i, top_indices],
+ height=bar_size,
+ label=label,
+ )
+ ax.set(
+ yticks=y_locs,
+ yticklabels=predictive_words,
+ ylim=[
+ 0 - 4 * bar_size,
+ len(top_indices) * (4 * bar_size + padding) - 4 * bar_size,
+ ],
)
- X_train = vectorizer.transform(data_train.data)
-else:
- vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5, stop_words="english")
- X_train = vectorizer.fit_transform(data_train.data)
-duration = time() - t0
-print("done in %fs at %0.3fMB/s" % (duration, data_train_size_mb / duration))
-print("n_samples: %d, n_features: %d" % X_train.shape)
+ ax.legend(loc="lower right")
+
+ print("top 5 keywords per class:")
+ print(top)
+
+ return ax
+
+
+_ = plot_feature_effects().set_title("Average feature effect on the original data")
# %%
-# Extracting features from the test data using the same vectorizer
-t0 = time()
-X_test = vectorizer.transform(data_test.data)
-duration = time() - t0
-print("done in %fs at %0.3fMB/s" % (duration, data_test_size_mb / duration))
-print("n_samples: %d, n_features: %d" % X_test.shape)
+# We can observe that the most predictive words are often strongly positively
+# associated with a single class and negatively associated with all the other
+# classes. Most of those positive associations are quite easy to interpret.
+# However, some words such as `"god"` and `"people"` are positively associated to
+# both `"talk.misc.religion"` and `"alt.atheism"` as those two classes expectedly
+# share some common vocabulary. Notice however that there are also words such as
+# `"christian"` and `"morality"` that are only positively associated with
+# `"talk.misc.religion"`. Furthermore, in this version of the dataset, the word
+# `"caltech"` is one of the top predictive features for atheism due to pollution
+# in the dataset coming from some sort of metadata such as the email addresses
+# of the sender of previous emails in the discussion as can be seen below:
+
+data_train = fetch_20newsgroups(
+ subset="train", categories=categories, shuffle=True, random_state=42
+)
+
+for doc in data_train.data:
+ if "caltech" in doc:
+ print(doc)
+ break
# %%
-# mapping from integer feature name to original token string
-if USE_HASHING:
- feature_names = None
-else:
- feature_names = vectorizer.get_feature_names_out()
+# Such headers, signature footers (and quoted metadata from previous messages)
+# can be considered side information that artificially reveals the newsgroup by
+# identifying the registered members and one would rather want our text
+# classifier to only learn from the "main content" of each text document instead
+# of relying on the leaked identity of the writers.
+#
+# Model with metadata stripping
+# -----------------------------
+#
+# The `remove` option of the 20 newsgroups dataset loader in scikit-learn allows
+# to heuristically attempt to filter out some of this unwanted metadata that
+# makes the classification problem artificially easier. Be aware that such
+# filtering of the text contents is far from perfect.
+#
+# Let us try to leverage this option to train a text classifier that does not
+# rely too much on this kind of metadata to make its decisions:
+(
+ X_train,
+ X_test,
+ y_train,
+ y_test,
+ feature_names,
+ target_names,
+) = load_dataset(remove=("headers", "footers", "quotes"))
+
+clf = RidgeClassifier(tol=1e-2, solver="sparse_cg")
+clf.fit(X_train, y_train)
+pred = clf.predict(X_test)
+
+fig, ax = plt.subplots(figsize=(10, 5))
+ConfusionMatrixDisplay.from_predictions(y_test, pred, ax=ax)
+ax.xaxis.set_ticklabels(target_names)
+ax.yaxis.set_ticklabels(target_names)
+_ = ax.set_title(
+ f"Confusion Matrix for {clf.__class__.__name__}\non filtered documents"
+)
# %%
-# Keeping only the best features
-from sklearn.feature_selection import SelectKBest, chi2
+# By looking at the confusion matrix, it is more evident that the scores of the
+# model trained with metadata were over-optimistic. The classification problem
+# without access to the metadata is less accurate but more representative of the
+# intended text classification problem.
-if SELECT_CHI2:
- print("Extracting %d best features by a chi-squared test" % SELECT_CHI2)
- t0 = time()
- ch2 = SelectKBest(chi2, k=SELECT_CHI2)
- X_train = ch2.fit_transform(X_train, y_train)
- X_test = ch2.transform(X_test)
- if feature_names is not None:
- # keep selected feature names
- feature_names = feature_names[ch2.get_support()]
- print("done in %fs" % (time() - t0))
- print()
+_ = plot_feature_effects().set_title("Average feature effects on filtered documents")
+# %%
+# In the next section we keep the dataset without metadata to compare several
+# classifiers.
# %%
-# Benchmark classifiers
-# ------------------------------------
+# Benchmarking classifiers
+# ========================
#
-# First we define small benchmarking utilities
-import numpy as np
-from sklearn import metrics
-from sklearn.utils.extmath import density
+# Scikit-learn provides many different kinds of classification algorithms. In
+# this section we will train a selection of those classifiers on the same text
+# classification problem and measure both their generalization performance
+# (accuracy on the test set) and their computation performance (speed), both at
+# training time and testing time. For such purpose we define the following
+# benchmarking utilities:
-
-def trim(s):
- """Trim string to fit on terminal (assuming 80-column display)"""
- return s if len(s) <= 80 else s[:77] + "..."
+from sklearn.utils.extmath import density
+from sklearn import metrics
-def benchmark(clf):
+def benchmark(clf, custom_name=False):
print("_" * 80)
print("Training: ")
print(clf)
t0 = time()
clf.fit(X_train, y_train)
train_time = time() - t0
- print("train time: %0.3fs" % train_time)
+ print(f"train time: {train_time:.3}s")
t0 = time()
pred = clf.predict(X_test)
test_time = time() - t0
- print("test time: %0.3fs" % test_time)
+ print(f"test time: {test_time:.3}s")
score = metrics.accuracy_score(y_test, pred)
- print("accuracy: %0.3f" % score)
+ print(f"accuracy: {score:.3}")
if hasattr(clf, "coef_"):
- print("dimensionality: %d" % clf.coef_.shape[1])
- print("density: %f" % density(clf.coef_))
-
- if feature_names is not None:
- print("top 10 keywords per class:")
- for i, label in enumerate(target_names):
- top10 = np.argsort(clf.coef_[i])[-10:]
- print(trim("%s: %s" % (label, " ".join(feature_names[top10]))))
+ print(f"dimensionality: {clf.coef_.shape[1]}")
+ print(f"density: {density(clf.coef_)}")
print()
- print("classification report:")
- print(metrics.classification_report(y_test, pred, target_names=target_names))
-
- print("confusion matrix:")
- print(metrics.confusion_matrix(y_test, pred))
-
print()
- clf_descr = str(clf).split("(")[0]
+ if custom_name:
+ clf_descr = str(custom_name)
+ else:
+ clf_descr = clf.__class__.__name__
return clf_descr, score, train_time, test_time
# %%
-# We now train and test the datasets with 15 different classification
-# models and get performance results for each model.
-from sklearn.feature_selection import SelectFromModel
-from sklearn.linear_model import RidgeClassifier
-from sklearn.pipeline import Pipeline
+# We now train and test the datasets with 8 different classification models and
+# get performance results for each model. The goal of this study is to highlight
+# the computation/accuracy tradeoffs of different types of classifiers for
+# such a multi-class text classification problem.
+#
+# Notice that the most important hyperparameters values were tuned using a grid
+# search procedure not shown in this notebook for the sake of simplicity.
+
+from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier
-from sklearn.linear_model import Perceptron
-from sklearn.linear_model import PassiveAggressiveClassifier
-from sklearn.naive_bayes import BernoulliNB, ComplementNB, MultinomialNB
+from sklearn.naive_bayes import ComplementNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.neighbors import NearestCentroid
from sklearn.ensemble import RandomForestClassifier
@@ -208,90 +370,84 @@ def benchmark(clf):
results = []
for clf, name in (
- (RidgeClassifier(tol=1e-2, solver="sag"), "Ridge Classifier"),
- (Perceptron(max_iter=50), "Perceptron"),
- (PassiveAggressiveClassifier(max_iter=50), "Passive-Aggressive"),
- (KNeighborsClassifier(n_neighbors=10), "kNN"),
- (RandomForestClassifier(), "Random forest"),
+ (LogisticRegression(C=5, max_iter=1000), "Logistic Regression"),
+ (RidgeClassifier(alpha=1.0, solver="sparse_cg"), "Ridge Classifier"),
+ (KNeighborsClassifier(n_neighbors=100), "kNN"),
+ (RandomForestClassifier(), "Random Forest"),
+ # L2 penalty Linear SVC
+ (LinearSVC(C=0.1, dual=False, max_iter=1000), "Linear SVC"),
+ # L2 penalty Linear SGD
+ (
+ SGDClassifier(
+ loss="log_loss", alpha=1e-4, n_iter_no_change=3, early_stopping=True
+ ),
+ "log-loss SGD",
+ ),
+ # NearestCentroid (aka Rocchio classifier)
+ (NearestCentroid(), "NearestCentroid"),
+ # Sparse naive Bayes classifier
+ (ComplementNB(alpha=0.1), "Complement naive Bayes"),
):
print("=" * 80)
print(name)
- results.append(benchmark(clf))
-
-for penalty in ["l2", "l1"]:
- print("=" * 80)
- print("%s penalty" % penalty.upper())
- # Train Liblinear model
- results.append(benchmark(LinearSVC(penalty=penalty, dual=False, tol=1e-3)))
-
- # Train SGD model
- results.append(benchmark(SGDClassifier(alpha=0.0001, max_iter=50, penalty=penalty)))
-
-# Train SGD with Elastic Net penalty
-print("=" * 80)
-print("Elastic-Net penalty")
-results.append(
- benchmark(SGDClassifier(alpha=0.0001, max_iter=50, penalty="elasticnet"))
-)
-
-# Train NearestCentroid without threshold
-print("=" * 80)
-print("NearestCentroid (aka Rocchio classifier)")
-results.append(benchmark(NearestCentroid()))
-
-# Train sparse Naive Bayes classifiers
-print("=" * 80)
-print("Naive Bayes")
-results.append(benchmark(MultinomialNB(alpha=0.01)))
-results.append(benchmark(BernoulliNB(alpha=0.01)))
-results.append(benchmark(ComplementNB(alpha=0.1)))
-
-print("=" * 80)
-print("LinearSVC with L1-based feature selection")
-# The smaller C, the stronger the regularization.
-# The more regularization, the more sparsity.
-results.append(
- benchmark(
- Pipeline(
- [
- (
- "feature_selection",
- SelectFromModel(LinearSVC(penalty="l1", dual=False, tol=1e-3)),
- ),
- ("classification", LinearSVC(penalty="l2")),
- ]
- )
- )
-)
-
+ results.append(benchmark(clf, name))
# %%
-# Add plots
-# ------------------------------------
-# The bar plot indicates the accuracy, training time (normalized) and test time
-# (normalized) of each classifier.
-import matplotlib.pyplot as plt
+# Plot accuracy, training and test time of each classifier
+# ========================================================
+#
+# The scatter plots show the trade-off between the test accuracy and the
+# training and testing time of each classifier.
indices = np.arange(len(results))
results = [[x[i] for x in results] for i in range(4)]
clf_names, score, training_time, test_time = results
-training_time = np.array(training_time) / np.max(training_time)
-test_time = np.array(test_time) / np.max(test_time)
-
-plt.figure(figsize=(12, 8))
-plt.title("Score")
-plt.barh(indices, score, 0.2, label="score", color="navy")
-plt.barh(indices + 0.3, training_time, 0.2, label="training time", color="c")
-plt.barh(indices + 0.6, test_time, 0.2, label="test time", color="darkorange")
-plt.yticks(())
-plt.legend(loc="best")
-plt.subplots_adjust(left=0.25)
-plt.subplots_adjust(top=0.95)
-plt.subplots_adjust(bottom=0.05)
-
-for i, c in zip(indices, clf_names):
- plt.text(-0.3, i, c)
-
-plt.show()
+training_time = np.array(training_time)
+test_time = np.array(test_time)
+
+fig, ax1 = plt.subplots(figsize=(10, 8))
+ax1.scatter(score, training_time, s=60)
+ax1.set(
+ title="Score-training time trade-off",
+ yscale="log",
+ xlabel="test accuracy",
+ ylabel="training time (s)",
+)
+fig, ax2 = plt.subplots(figsize=(10, 8))
+ax2.scatter(score, test_time, s=60)
+ax2.set(
+ title="Score-test time trade-off",
+ yscale="log",
+ xlabel="test accuracy",
+ ylabel="test time (s)",
+)
+
+for i, txt in enumerate(clf_names):
+ ax1.annotate(txt, (score[i], training_time[i]))
+ ax2.annotate(txt, (score[i], test_time[i]))
+
+# %%
+# The naive Bayes model has the best trade-off between score and
+# training/testing time, while Random Forest is both slow to train, expensive to
+# predict and has a comparatively bad accuracy. This is expected: for
+# high-dimensional prediction problems, linear models are often better suited as
+# most problems become linearly separable when the feature space has 10,000
+# dimensions or more.
+#
+# The difference in training speed and accuracy of the linear models can be
+# explained by the choice of the loss function they optimize and the kind of
+# regularization they use. Be aware that some linear models with the same loss
+# but a different solver or regularization configuration may yield different
+# fitting times and test accuracy. We can observe on the second plot that once
+# trained, all linear models have approximately the same prediction speed which
+# is expected because they all implement the same prediction function.
+#
+# KNeighborsClassifier has a relatively low accuracy and has the highest testing
+# time. The long prediction time is also expected: for each prediction the model
+# has to compute the pairwise distances between the testing sample and each
+# document in the training set, which is computationally expensive. Furthermore,
+# the "curse of dimensionality" harms the ability of this model to yield
+# competitive accuracy in the high dimensional feature space of text
+# classification problems.
diff --git a/examples/text/plot_document_clustering.py b/examples/text/plot_document_clustering.py
index 66b25ec9851de..04446fd82f964 100644
--- a/examples/text/plot_document_clustering.py
+++ b/examples/text/plot_document_clustering.py
@@ -3,284 +3,448 @@
Clustering text documents using k-means
=======================================
-This is an example showing how the scikit-learn can be used to cluster
-documents by topics using a bag-of-words approach. This example uses
-a scipy.sparse matrix to store the features instead of standard numpy arrays.
-
-Two feature extraction methods can be used in this example:
-
- - TfidfVectorizer uses a in-memory vocabulary (a python dict) to map the most
- frequent words to features indices and hence compute a word occurrence
- frequency (sparse) matrix. The word frequencies are then reweighted using
- the Inverse Document Frequency (IDF) vector collected feature-wise over
- the corpus.
-
- - HashingVectorizer hashes word occurrences to a fixed dimensional space,
- possibly with collisions. The word count vectors are then normalized to
- each have l2-norm equal to one (projected to the euclidean unit-ball) which
- seems to be important for k-means to work in high dimensional space.
-
- HashingVectorizer does not provide IDF weighting as this is a stateless
- model (the fit method does nothing). When IDF weighting is needed it can
- be added by pipelining its output to a TfidfTransformer instance.
-
-Two algorithms are demoed: ordinary k-means and its more scalable cousin
-minibatch k-means.
-
-Additionally, latent semantic analysis can also be used to reduce
-dimensionality and discover latent patterns in the data.
-
-It can be noted that k-means (and minibatch k-means) are very sensitive to
-feature scaling and that in this case the IDF weighting helps improve the
-quality of the clustering by quite a lot as measured against the "ground truth"
-provided by the class label assignments of the 20 newsgroups dataset.
-
-This improvement is not visible in the Silhouette Coefficient which is small
-for both as this measure seem to suffer from the phenomenon called
-"Concentration of Measure" or "Curse of Dimensionality" for high dimensional
-datasets such as text data. Other measures such as V-measure and Adjusted Rand
-Index are information theoretic based evaluation scores: as they are only based
-on cluster assignments rather than distances, hence not affected by the curse
-of dimensionality.
-
-Note: as k-means is optimizing a non-convex objective function, it will likely
-end up in a local optimum. Several runs with independent random init might be
-necessary to get a good convergence.
+This is an example showing how the scikit-learn API can be used to cluster
+documents by topics using a `Bag of Words approach
+`_.
+
+Two algorithms are demoed: :class:`~sklearn.cluster.KMeans` and its more
+scalable variant, :class:`~sklearn.cluster.MiniBatchKMeans`. Additionally,
+latent semantic analysis is used to reduce dimensionality and discover latent
+patterns in the data.
+
+This example uses two different text vectorizers: a
+:class:`~sklearn.feature_extraction.text.TfidfVectorizer` and a
+:class:`~sklearn.feature_extraction.text.HashingVectorizer`. See the example
+notebook :ref:`sphx_glr_auto_examples_text_plot_hashing_vs_dict_vectorizer.py`
+for more information on vectorizers and a comparison of their processing times.
+
+For document analysis via a supervised learning approach, see the example script
+:ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`.
"""
# Author: Peter Prettenhofer
# Lars Buitinck
+# Olivier Grisel
+# Arturo Amor
# License: BSD 3 clause
-from sklearn.datasets import fetch_20newsgroups
-from sklearn.decomposition import TruncatedSVD
-from sklearn.feature_extraction.text import TfidfVectorizer
-from sklearn.feature_extraction.text import HashingVectorizer
-from sklearn.feature_extraction.text import TfidfTransformer
-from sklearn.pipeline import make_pipeline
-from sklearn.preprocessing import Normalizer
-from sklearn import metrics
-
-from sklearn.cluster import KMeans, MiniBatchKMeans
-
-import logging
-from optparse import OptionParser
-import sys
-from time import time
+# %%
+# Loading text data
+# =================
+#
+# We load data from :ref:`20newsgroups_dataset`, which comprises around 18,000
+# newsgroups posts on 20 topics. For illustrative purposes and to reduce the
+# computational cost, we select a subset of 4 topics only accounting for around
+# 3,400 documents. See the example
+# :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`
+# to gain intuition on the overlap of such topics.
+#
+# Notice that, by default, the text samples contain some message metadata such
+# as `"headers"`, `"footers"` (signatures) and `"quotes"` to other posts. We use
+# the `remove` parameter from :func:`~sklearn.datasets.fetch_20newsgroups` to
+# strip those features and have a more sensible clustering problem.
import numpy as np
+from sklearn.datasets import fetch_20newsgroups
+categories = [
+ "alt.atheism",
+ "talk.religion.misc",
+ "comp.graphics",
+ "sci.space",
+]
-# Display progress logs on stdout
-logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
-
-# parse commandline arguments
-op = OptionParser()
-op.add_option(
- "--lsa",
- dest="n_components",
- type="int",
- help="Preprocess documents with latent semantic analysis.",
-)
-op.add_option(
- "--no-minibatch",
- action="store_false",
- dest="minibatch",
- default=True,
- help="Use ordinary k-means algorithm (in batch mode).",
-)
-op.add_option(
- "--no-idf",
- action="store_false",
- dest="use_idf",
- default=True,
- help="Disable Inverse Document Frequency feature weighting.",
-)
-op.add_option(
- "--use-hashing",
- action="store_true",
- default=False,
- help="Use a hashing feature vectorizer",
-)
-op.add_option(
- "--n-features",
- type=int,
- default=10000,
- help="Maximum number of features (dimensions) to extract from text.",
-)
-op.add_option(
- "--verbose",
- action="store_true",
- dest="verbose",
- default=False,
- help="Print progress reports inside k-means algorithm.",
+dataset = fetch_20newsgroups(
+ remove=("headers", "footers", "quotes"),
+ subset="all",
+ categories=categories,
+ shuffle=True,
+ random_state=42,
)
-print(__doc__)
+labels = dataset.target
+unique_labels, category_sizes = np.unique(labels, return_counts=True)
+true_k = unique_labels.shape[0]
+print(f"{len(dataset.data)} documents - {true_k} categories")
-def is_interactive():
- return not hasattr(sys.modules["__main__"], "__file__")
+# %%
+# Quantifying the quality of clustering results
+# =============================================
+#
+# In this section we define a function to score different clustering pipelines
+# using several metrics.
+#
+# Clustering algorithms are fundamentally unsupervised learning methods.
+# However, since we happen to have class labels for this specific dataset, it is
+# possible to use evaluation metrics that leverage this "supervised" ground
+# truth information to quantify the quality of the resulting clusters. Examples
+# of such metrics are the following:
+#
+# - homogeneity, which quantifies how much clusters contain only members of a
+# single class;
+#
+# - completeness, which quantifies how much members of a given class are
+# assigned to the same clusters;
+#
+# - V-measure, the harmonic mean of completeness and homogeneity;
+#
+# - Rand-Index, which measures how frequently pairs of data points are grouped
+# consistently according to the result of the clustering algorithm and the
+# ground truth class assignment;
+#
+# - Adjusted Rand-Index, a chance-adjusted Rand-Index such that random cluster
+# assignment have an ARI of 0.0 in expectation.
+#
+# If the ground truth labels are not known, evaluation can only be performed
+# using the model results itself. In that case, the Silhouette Coefficient comes
+# in handy.
+#
+# For more reference, see :ref:`clustering_evaluation`.
+
+from collections import defaultdict
+from sklearn import metrics
+from time import time
+evaluations = []
+evaluations_std = []
+
+
+def fit_and_evaluate(km, X, name=None, n_runs=5):
+ name = km.__class__.__name__ if name is None else name
+
+ train_times = []
+ scores = defaultdict(list)
+ for seed in range(n_runs):
+ km.set_params(random_state=seed)
+ t0 = time()
+ km.fit(X)
+ train_times.append(time() - t0)
+ scores["Homogeneity"].append(metrics.homogeneity_score(labels, km.labels_))
+ scores["Completeness"].append(metrics.completeness_score(labels, km.labels_))
+ scores["V-measure"].append(metrics.v_measure_score(labels, km.labels_))
+ scores["Adjusted Rand-Index"].append(
+ metrics.adjusted_rand_score(labels, km.labels_)
+ )
+ scores["Silhouette Coefficient"].append(
+ metrics.silhouette_score(X, km.labels_, sample_size=2000)
+ )
+ train_times = np.asarray(train_times)
+
+ print(f"clustering done in {train_times.mean():.2f} ± {train_times.std():.2f} s ")
+ evaluation = {
+ "estimator": name,
+ "train_time": train_times.mean(),
+ }
+ evaluation_std = {
+ "estimator": name,
+ "train_time": train_times.std(),
+ }
+ for score_name, score_values in scores.items():
+ mean_score, std_score = np.mean(score_values), np.std(score_values)
+ print(f"{score_name}: {mean_score:.3f} ± {std_score:.3f}")
+ evaluation[score_name] = mean_score
+ evaluation_std[score_name] = std_score
+ evaluations.append(evaluation)
+ evaluations_std.append(evaluation_std)
-if not is_interactive():
- op.print_help()
- print()
-# work-around for Jupyter notebook and IPython console
-argv = [] if is_interactive() else sys.argv[1:]
-(opts, args) = op.parse_args(argv)
-if len(args) > 0:
- op.error("this script takes no arguments.")
- sys.exit(1)
+# %%
+# K-means clustering on text features
+# ===================================
+#
+# Two feature extraction methods are used in this example:
+#
+# - :class:`~sklearn.feature_extraction.text.TfidfVectorizer` uses an in-memory
+# vocabulary (a Python dict) to map the most frequent words to features
+# indices and hence compute a word occurrence frequency (sparse) matrix. The
+# word frequencies are then reweighted using the Inverse Document Frequency
+# (IDF) vector collected feature-wise over the corpus.
+#
+# - :class:`~sklearn.feature_extraction.text.HashingVectorizer` hashes word
+# occurrences to a fixed dimensional space, possibly with collisions. The word
+# count vectors are then normalized to each have l2-norm equal to one
+# (projected to the euclidean unit-sphere) which seems to be important for
+# k-means to work in high dimensional space.
+#
+# Furthermore it is possible to post-process those extracted features using
+# dimensionality reduction. We will explore the impact of those choices on the
+# clustering quality in the following.
+#
+# Feature Extraction using TfidfVectorizer
+# ----------------------------------------
+#
+# We first benchmark the estimators using a dictionary vectorizer along with an
+# IDF normalization as provided by
+# :class:`~sklearn.feature_extraction.text.TfidfVectorizer`.
+from sklearn.feature_extraction.text import TfidfVectorizer
-# %%
-# Load some categories from the training set
-# ------------------------------------------
+vectorizer = TfidfVectorizer(
+ max_df=0.5,
+ min_df=5,
+ stop_words="english",
+)
+t0 = time()
+X_tfidf = vectorizer.fit_transform(dataset.data)
-categories = [
- "alt.atheism",
- "talk.religion.misc",
- "comp.graphics",
- "sci.space",
-]
-# Uncomment the following to do the analysis on all the categories
-# categories = None
+print(f"vectorization done in {time() - t0:.3f} s")
+print(f"n_samples: {X_tfidf.shape[0]}, n_features: {X_tfidf.shape[1]}")
-print("Loading 20 newsgroups dataset for categories:")
-print(categories)
+# %%
+# After ignoring terms that appear in more than 50% of the documents (as set by
+# `max_df=0.5`) and terms that are not present in at least 5 documents (set by
+# `min_df=5`), the resulting number of unique terms `n_features` is around
+# 8,000. We can additionally quantify the sparsity of the `X_tfidf` matrix as
+# the fraction of non-zero entries devided by the total number of elements.
-dataset = fetch_20newsgroups(
- subset="all", categories=categories, shuffle=True, random_state=42
-)
+print(f"{X_tfidf.nnz / np.prod(X_tfidf.shape):.3f}")
-print("%d documents" % len(dataset.data))
-print("%d categories" % len(dataset.target_names))
+# %%
+# We find that around 0.7% of the entries of the `X_tfidf` matrix are non-zero.
+#
+# .. _kmeans_sparse_high_dim:
+#
+# Clustering sparse data with k-means
+# -----------------------------------
+#
+# As both :class:`~sklearn.cluster.KMeans` and
+# :class:`~sklearn.cluster.MiniBatchKMeans` optimize a non-convex objective
+# function, their clustering is not guaranteed to be optimal for a given random
+# init. Even further, on sparse high-dimensional data such as text vectorized
+# using the Bag of Words approach, k-means can initialize centroids on extremely
+# isolated data points. Those data points can stay their own centroids all
+# along.
+#
+# The following code illustrates how the previous phenomenon can sometimes lead
+# to highly imbalanced clusters, depending on the random initialization:
+
+from sklearn.cluster import KMeans
+
+for seed in range(5):
+ kmeans = KMeans(
+ n_clusters=true_k,
+ max_iter=100,
+ n_init=1,
+ random_state=seed,
+ ).fit(X_tfidf)
+ cluster_ids, cluster_sizes = np.unique(kmeans.labels_, return_counts=True)
+ print(f"Number of elements asigned to each cluster: {cluster_sizes}")
print()
+print(
+ "True number of documents in each category according to the class labels: "
+ f"{category_sizes}"
+)
+
+# %%
+# To avoid this problem, one possibility is to increase the number of runs with
+# independent random initiations `n_init`. In such case the clustering with the
+# best inertia (objective function of k-means) is chosen.
+
+kmeans = KMeans(
+ n_clusters=true_k,
+ max_iter=100,
+ n_init=5,
+)
+fit_and_evaluate(kmeans, X_tfidf, name="KMeans\non tf-idf vectors")
# %%
-# Feature Extraction
-# ------------------
+# All those clustering evaluation metrics have a maximum value of 1.0 (for a
+# perfect clustering result). Higher values are better. Values of the Adjusted
+# Rand-Index close to 0.0 correspond to a random labeling. Notice from the
+# scores above that the cluster assignment is indeed well above chance level,
+# but the overall quality can certainly improve.
+#
+# Keep in mind that the class labels may not reflect accurately the document
+# topics and therefore metrics that use labels are not necessarily the best to
+# evaluate the quality of our clustering pipeline.
+#
+# Performing dimensionality reduction using LSA
+# ---------------------------------------------
+#
+# A `n_init=1` can still be used as long as the dimension of the vectorized
+# space is reduced first to make k-means more stable. For such purpose we use
+# :class:`~sklearn.decomposition.TruncatedSVD`, which works on term count/tf-idf
+# matrices. Since SVD results are not normalized, we redo the normalization to
+# improve the :class:`~sklearn.cluster.KMeans` result. Using SVD to reduce the
+# dimensionality of TF-IDF document vectors is often known as `latent semantic
+# analysis `_ (LSA) in
+# the information retrieval and text mining literature.
-labels = dataset.target
-true_k = np.unique(labels).shape[0]
+from sklearn.decomposition import TruncatedSVD
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import Normalizer
-print("Extracting features from the training dataset using a sparse vectorizer")
+
+lsa = make_pipeline(TruncatedSVD(n_components=100), Normalizer(copy=False))
t0 = time()
-if opts.use_hashing:
- if opts.use_idf:
- # Perform an IDF normalization on the output of HashingVectorizer
- hasher = HashingVectorizer(
- n_features=opts.n_features,
- stop_words="english",
- alternate_sign=False,
- norm=None,
- )
- vectorizer = make_pipeline(hasher, TfidfTransformer())
- else:
- vectorizer = HashingVectorizer(
- n_features=opts.n_features,
- stop_words="english",
- alternate_sign=False,
- norm="l2",
- )
-else:
- vectorizer = TfidfVectorizer(
- max_df=0.5,
- max_features=opts.n_features,
- min_df=2,
- stop_words="english",
- use_idf=opts.use_idf,
- )
-X = vectorizer.fit_transform(dataset.data)
-
-print("done in %fs" % (time() - t0))
-print("n_samples: %d, n_features: %d" % X.shape)
-print()
+X_lsa = lsa.fit_transform(X_tfidf)
+explained_variance = lsa[0].explained_variance_ratio_.sum()
-if opts.n_components:
- print("Performing dimensionality reduction using LSA")
- t0 = time()
- # Since LSA/SVD results are not normalized,
- # we redo the normalization to improve the k-means result.
- svd = TruncatedSVD(opts.n_components)
- normalizer = Normalizer(copy=False)
- lsa = make_pipeline(svd, normalizer)
+print(f"LSA done in {time() - t0:.3f} s")
+print(f"Explained variance of the SVD step: {explained_variance * 100:.1f}%")
- X = lsa.fit_transform(X)
+# %%
+# Using a single initialization means the processing time will be reduced for
+# both :class:`~sklearn.cluster.KMeans` and
+# :class:`~sklearn.cluster.MiniBatchKMeans`.
+
+kmeans = KMeans(
+ n_clusters=true_k,
+ max_iter=100,
+ n_init=1,
+)
- print("done in %fs" % (time() - t0))
+fit_and_evaluate(kmeans, X_lsa, name="KMeans\nwith LSA on tf-idf vectors")
- explained_variance = svd.explained_variance_ratio_.sum()
- print(
- "Explained variance of the SVD step: {}%".format(int(explained_variance * 100))
- )
+# %%
+# We can observe that clustering on the LSA representation of the document is
+# significantly faster (both because of `n_init=1` and because the
+# dimensionality of the LSA feature space is much smaller). Furthermore, all the
+# clustering evaluation metrics have improved. We repeat the experiment with
+# :class:`~sklearn.cluster.MiniBatchKMeans`.
+
+from sklearn.cluster import MiniBatchKMeans
+
+minibatch_kmeans = MiniBatchKMeans(
+ n_clusters=true_k,
+ n_init=1,
+ init_size=1000,
+ batch_size=1000,
+)
- print()
+fit_and_evaluate(
+ minibatch_kmeans,
+ X_lsa,
+ name="MiniBatchKMeans\nwith LSA on tf-idf vectors",
+)
+# %%
+# Top terms per cluster
+# ---------------------
+#
+# Since :class:`~sklearn.feature_extraction.text.TfidfVectorizer` can be
+# inverted we can identify the cluster centers, which provide an intuition of
+# the most influential words **for each cluster**. See the example script
+# :ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`
+# for a comparison with the most predictive words **for each target class**.
+
+original_space_centroids = lsa[0].inverse_transform(kmeans.cluster_centers_)
+order_centroids = original_space_centroids.argsort()[:, ::-1]
+terms = vectorizer.get_feature_names_out()
+
+for i in range(true_k):
+ print(f"Cluster {i}: ", end="")
+ for ind in order_centroids[i, :10]:
+ print(f"{terms[ind]} ", end="")
+ print()
# %%
-# Clustering
-# ----------
+# HashingVectorizer
+# -----------------
+# An alternative vectorization can be done using a
+# :class:`~sklearn.feature_extraction.text.HashingVectorizer` instance, which
+# does not provide IDF weighting as this is a stateless model (the fit method
+# does nothing). When IDF weighting is needed it can be added by pipelining the
+# :class:`~sklearn.feature_extraction.text.HashingVectorizer` output to a
+# :class:`~sklearn.feature_extraction.text.TfidfTransformer` instance. In this
+# case we also add LSA to the pipeline to reduce the dimension and sparcity of
+# the hashed vector space.
-if opts.minibatch:
- km = MiniBatchKMeans(
- n_clusters=true_k,
- init="k-means++",
- n_init=1,
- init_size=1000,
- batch_size=1000,
- verbose=opts.verbose,
- )
-else:
- km = KMeans(
- n_clusters=true_k,
- init="k-means++",
- max_iter=100,
- n_init=1,
- verbose=opts.verbose,
- )
+from sklearn.feature_extraction.text import HashingVectorizer
+from sklearn.feature_extraction.text import TfidfTransformer
-print("Clustering sparse data with %s" % km)
-t0 = time()
-km.fit(X)
-print("done in %0.3fs" % (time() - t0))
-print()
+lsa_vectorizer = make_pipeline(
+ HashingVectorizer(stop_words="english", n_features=50_000),
+ TfidfTransformer(),
+ TruncatedSVD(n_components=100, random_state=0),
+ Normalizer(copy=False),
+)
+t0 = time()
+X_hashed_lsa = lsa_vectorizer.fit_transform(dataset.data)
+print(f"vectorization done in {time() - t0:.3f} s")
# %%
-# Performance metrics
-# -------------------
+# One can observe that the LSA step takes a relatively long time to fit,
+# especially with hashed vectors. The reason is that a hashed space is typically
+# large (set to `n_features=50_000` in this example). One can try lowering the
+# number of features at the expense of having a larger fraction of features with
+# hash collisions as shown in the example notebook
+# :ref:`sphx_glr_auto_examples_text_plot_hashing_vs_dict_vectorizer.py`.
+#
+# We now fit and evaluate the `kmeans` and `minibatch_kmeans` instances on this
+# hashed-lsa-reduced data:
+
+fit_and_evaluate(kmeans, X_hashed_lsa, name="KMeans\nwith LSA on hashed vectors")
-print("Homogeneity: %0.3f" % metrics.homogeneity_score(labels, km.labels_))
-print("Completeness: %0.3f" % metrics.completeness_score(labels, km.labels_))
-print("V-measure: %0.3f" % metrics.v_measure_score(labels, km.labels_))
-print("Adjusted Rand-Index: %.3f" % metrics.adjusted_rand_score(labels, km.labels_))
-print(
- "Silhouette Coefficient: %0.3f"
- % metrics.silhouette_score(X, km.labels_, sample_size=1000)
+# %%
+fit_and_evaluate(
+ minibatch_kmeans,
+ X_hashed_lsa,
+ name="MiniBatchKMeans\nwith LSA on hashed vectors",
)
-print()
+# %%
+# Both methods lead to good results that are similar to running the same models
+# on the traditional LSA vectors (without hashing).
+#
+# Clustering evaluation summary
+# ==============================
+import pandas as pd
+import matplotlib.pyplot as plt
-# %%
+fig, (ax0, ax1) = plt.subplots(ncols=2, figsize=(16, 6), sharey=True)
+
+df = pd.DataFrame(evaluations[::-1]).set_index("estimator")
+df_std = pd.DataFrame(evaluations_std[::-1]).set_index("estimator")
+
+df.drop(
+ ["train_time"],
+ axis="columns",
+).plot.barh(ax=ax0, xerr=df_std)
+ax0.set_xlabel("Clustering scores")
+ax0.set_ylabel("")
-if not opts.use_hashing:
- print("Top terms per cluster:")
-
- if opts.n_components:
- original_space_centroids = svd.inverse_transform(km.cluster_centers_)
- order_centroids = original_space_centroids.argsort()[:, ::-1]
- else:
- order_centroids = km.cluster_centers_.argsort()[:, ::-1]
-
- terms = vectorizer.get_feature_names_out()
- for i in range(true_k):
- print("Cluster %d:" % i, end="")
- for ind in order_centroids[i, :10]:
- print(" %s" % terms[ind], end="")
- print()
+df["train_time"].plot.barh(ax=ax1, xerr=df_std["train_time"])
+ax1.set_xlabel("Clustering time (s)")
+plt.tight_layout()
+
+# %%
+# :class:`~sklearn.cluster.KMeans` and :class:`~sklearn.cluster.MiniBatchKMeans`
+# suffer from the phenomenon called the `Curse of Dimensionality
+# `_ for high dimensional
+# datasets such as text data. That is the reason why the overall scores improve
+# when using LSA. Using LSA reduced data also improves the stability and
+# requires lower clustering time, though keep in mind that the LSA step itself
+# takes a long time, especially with hashed vectors.
+#
+# The Silhouette Coefficient is defined between 0 and 1. In all cases we obtain
+# values close to 0 (even if they improve a bit after using LSA) because its
+# definition requires measuring distances, in contrast with other evaluation
+# metrics such as the V-measure and the Adjusted Rand Index which are only based
+# on cluster assignments rather than distances. Notice that strictly speaking,
+# one should not compare the Silhouette Coefficient between spaces of different
+# dimension, due to the different notions of distance they imply.
+#
+# The homogeneity, completeness and hence v-measure metrics do not yield a
+# baseline with regards to random labeling: this means that depending on the
+# number of samples, clusters and ground truth classes, a completely random
+# labeling will not always yield the same values. In particular random labeling
+# won't yield zero scores, especially when the number of clusters is large. This
+# problem can safely be ignored when the number of samples is more than a
+# thousand and the number of clusters is less than 10, which is the case of the
+# present example. For smaller sample sizes or larger number of clusters it is
+# safer to use an adjusted index such as the Adjusted Rand Index (ARI). See the
+# example
+# :ref:`sphx_glr_auto_examples_cluster_plot_adjusted_for_chance_measures.py` for
+# a demo on the effect of random labeling.
+#
+# The size of the error bars show that :class:`~sklearn.cluster.MiniBatchKMeans`
+# is less stable than :class:`~sklearn.cluster.KMeans` for this relatively small
+# dataset. It is more interesting to use when the number of samples is much
+# bigger, but it can come at the expense of a small degradation in clustering
+# quality compared to the traditional k-means algorithm.
diff --git a/examples/text/plot_hashing_vs_dict_vectorizer.py b/examples/text/plot_hashing_vs_dict_vectorizer.py
index ce359cd137487..8200c646f69ee 100644
--- a/examples/text/plot_hashing_vs_dict_vectorizer.py
+++ b/examples/text/plot_hashing_vs_dict_vectorizer.py
@@ -3,109 +3,382 @@
FeatureHasher and DictVectorizer Comparison
===========================================
-Compares FeatureHasher and DictVectorizer by using both to vectorize
-text documents.
+In this example we illustrate text vectorization, which is the process of
+representing non-numerical input data (such as dictionaries or text documents)
+as vectors of real numbers.
-The example demonstrates syntax and speed only; it doesn't actually do
-anything useful with the extracted vectors. See the example scripts
-{document_classification_20newsgroups,clustering}.py for actual learning
-on text documents.
+We first compare :func:`~sklearn.feature_extraction.FeatureHasher` and
+:func:`~sklearn.feature_extraction.DictVectorizer` by using both methods to
+vectorize text documents that are preprocessed (tokenized) with the help of a
+custom Python function.
-A discrepancy between the number of terms reported for DictVectorizer and
-for FeatureHasher is to be expected due to hash collisions.
+Later we introduce and analyze the text-specific vectorizers
+:func:`~sklearn.feature_extraction.text.HashingVectorizer`,
+:func:`~sklearn.feature_extraction.text.CountVectorizer` and
+:func:`~sklearn.feature_extraction.text.TfidfVectorizer` that handle both the
+tokenization and the assembling of the feature matrix within a single class.
+
+The objective of the example is to demonstrate the usage of text vectorization
+API and to compare their processing time. See the example scripts
+:ref:`sphx_glr_auto_examples_text_plot_document_classification_20newsgroups.py`
+and :ref:`sphx_glr_auto_examples_text_plot_document_clustering.py` for actual
+learning on text documents.
"""
# Author: Lars Buitinck
+# Olivier Grisel
+# Arturo Amor
# License: BSD 3 clause
-from collections import defaultdict
-import re
-import sys
-from time import time
-
-import numpy as np
+# %%
+# Load Data
+# ---------
+#
+# We load data from :ref:`20newsgroups_dataset`, which comprises around
+# 18000 newsgroups posts on 20 topics split in two subsets: one for training and
+# one for testing. For the sake of simplicity and reducing the computational
+# cost, we select a subset of 7 topics and use the training set only.
from sklearn.datasets import fetch_20newsgroups
-from sklearn.feature_extraction import DictVectorizer, FeatureHasher
+categories = [
+ "alt.atheism",
+ "comp.graphics",
+ "comp.sys.ibm.pc.hardware",
+ "misc.forsale",
+ "rec.autos",
+ "sci.space",
+ "talk.religion.misc",
+]
+
+print("Loading 20 newsgroups training data")
+raw_data, _ = fetch_20newsgroups(subset="train", categories=categories, return_X_y=True)
+data_size_mb = sum(len(s.encode("utf-8")) for s in raw_data) / 1e6
+print(f"{len(raw_data)} documents - {data_size_mb:.3f}MB")
-def n_nonzero_columns(X):
- """Returns the number of non-zero columns in a CSR matrix X."""
- return len(np.unique(X.nonzero()[1]))
+# %%
+# Define preprocessing functions
+# ------------------------------
+#
+# A token may be a word, part of a word or anything comprised between spaces or
+# symbols in a string. Here we define a function that extracts the tokens using
+# a simple regular expression (regex) that matches Unicode word characters. This
+# includes most characters that can be part of a word in any language, as well
+# as numbers and the underscore:
+
+import re
-def tokens(doc):
+def tokenize(doc):
"""Extract tokens from doc.
- This uses a simple regex to break strings into tokens. For a more
- principled approach, see CountVectorizer or TfidfVectorizer.
+ This uses a simple regex that matches word characters to break strings
+ into tokens. For a more principled approach, see CountVectorizer or
+ TfidfVectorizer.
"""
return (tok.lower() for tok in re.findall(r"\w+", doc))
+list(tokenize("This is a simple example, isn't it?"))
+
+# %%
+# We define an additional function that counts the (frequency of) occurrence of
+# each token in a given document. It returns a frequency dictionary to be used
+# by the vectorizers.
+
+from collections import defaultdict
+
+
def token_freqs(doc):
- """Extract a dict mapping tokens from doc to their frequencies."""
+ """Extract a dict mapping tokens from doc to their occurrences."""
+
freq = defaultdict(int)
- for tok in tokens(doc):
+ for tok in tokenize(doc):
freq[tok] += 1
return freq
-categories = [
- "alt.atheism",
- "comp.graphics",
- "comp.sys.ibm.pc.hardware",
- "misc.forsale",
- "rec.autos",
- "sci.space",
- "talk.religion.misc",
-]
-# Uncomment the following line to use a larger set (11k+ documents)
-# categories = None
+token_freqs("That is one example, but this is another one")
-print(__doc__)
-print("Usage: %s [n_features_for_hashing]" % sys.argv[0])
-print(" The default number of features is 2**18.")
-print()
+# %%
+# Observe in particular that the repeated token `"is"` is counted twice for
+# instance.
+#
+# Breaking a text document into word tokens, potentially losing the order
+# information between the words in a sentence is often called a `Bag of Words
+# representation `_.
-try:
- n_features = int(sys.argv[1])
-except IndexError:
- n_features = 2**18
-except ValueError:
- print("not a valid number of features: %r" % sys.argv[1])
- sys.exit(1)
+# %%
+# DictVectorizer
+# --------------
+#
+# First we benchmark the :func:`~sklearn.feature_extraction.DictVectorizer`,
+# then we compare it to :func:`~sklearn.feature_extraction.FeatureHasher` as
+# both of them receive dictionaries as input.
+from time import time
+from sklearn.feature_extraction import DictVectorizer
-print("Loading 20 newsgroups training data")
-raw_data, _ = fetch_20newsgroups(subset="train", categories=categories, return_X_y=True)
-data_size_mb = sum(len(s.encode("utf-8")) for s in raw_data) / 1e6
-print("%d documents - %0.3fMB" % (len(raw_data), data_size_mb))
-print()
+dict_count_vectorizers = defaultdict(list)
-print("DictVectorizer")
t0 = time()
vectorizer = DictVectorizer()
vectorizer.fit_transform(token_freqs(d) for d in raw_data)
duration = time() - t0
-print("done in %fs at %0.3fMB/s" % (duration, data_size_mb / duration))
-print("Found %d unique terms" % len(vectorizer.get_feature_names_out()))
-print()
+dict_count_vectorizers["vectorizer"].append(
+ vectorizer.__class__.__name__ + "\non freq dicts"
+)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {len(vectorizer.get_feature_names_out())} unique terms")
+
+# %%
+# The actual mapping from text token to column index is explicitly stored in
+# the `.vocabulary_` attribute which is a potentially very large Python
+# dictionary:
+type(vectorizer.vocabulary_)
+
+# %%
+len(vectorizer.vocabulary_)
+
+# %%
+vectorizer.vocabulary_["example"]
+
+# %%
+# FeatureHasher
+# -------------
+#
+# Dictionaries take up a large amount of storage space and grow in size as the
+# training set grows. Instead of growing the vectors along with a dictionary,
+# feature hashing builds a vector of pre-defined length by applying a hash
+# function `h` to the features (e.g., tokens), then using the hash values
+# directly as feature indices and updating the resulting vector at those
+# indices. When the feature space is not large enough, hashing functions tend to
+# map distinct values to the same hash code (hash collisions). As a result, it
+# is impossible to determine what object generated any particular hash code.
+#
+# Because of the above it is impossible to recover the original tokens from the
+# feature matrix and the best approach to estimate the number of unique terms in
+# the original dictionary is to count the number of active columns in the
+# encoded feature matrix. For such a purpose we define the following function:
+
+import numpy as np
+
+
+def n_nonzero_columns(X):
+ """Number of columns with at least one non-zero value in a CSR matrix.
+
+ This is useful to count the number of features columns that are effectively
+ active when using the FeatureHasher.
+ """
+ return len(np.unique(X.nonzero()[1]))
+
+
+# %%
+# The default number of features for the
+# :func:`~sklearn.feature_extraction.FeatureHasher` is 2**20. Here we set
+# `n_features = 2**18` to illustrate hash collisions.
+#
+# **FeatureHasher on frequency dictionaries**
+
+from sklearn.feature_extraction import FeatureHasher
-print("FeatureHasher on frequency dicts")
t0 = time()
-hasher = FeatureHasher(n_features=n_features)
+hasher = FeatureHasher(n_features=2**18)
X = hasher.transform(token_freqs(d) for d in raw_data)
duration = time() - t0
-print("done in %fs at %0.3fMB/s" % (duration, data_size_mb / duration))
-print("Found %d unique terms" % n_nonzero_columns(X))
-print()
+dict_count_vectorizers["vectorizer"].append(
+ hasher.__class__.__name__ + "\non freq dicts"
+)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {n_nonzero_columns(X)} unique tokens")
+
+# %%
+# The number of unique tokens when using the
+# :func:`~sklearn.feature_extraction.FeatureHasher` is lower than those obtained
+# using the :func:`~sklearn.feature_extraction.DictVectorizer`. This is due to
+# hash collisions.
+#
+# The number of collisions can be reduced by increasing the feature space.
+# Notice that the speed of the vectorizer does not change significantly when
+# setting a large number of features, though it causes larger coefficient
+# dimensions and then requires more memory usage to store them, even if a
+# majority of them is inactive.
-print("FeatureHasher on raw tokens")
t0 = time()
-hasher = FeatureHasher(n_features=n_features, input_type="string")
-X = hasher.transform(tokens(d) for d in raw_data)
+hasher = FeatureHasher(n_features=2**22)
+X = hasher.transform(token_freqs(d) for d in raw_data)
duration = time() - t0
-print("done in %fs at %0.3fMB/s" % (duration, data_size_mb / duration))
-print("Found %d unique terms" % n_nonzero_columns(X))
+
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {n_nonzero_columns(X)} unique tokens")
+
+# %%
+# We confirm that the number of unique tokens gets closer to the number of
+# unique terms found by the :func:`~sklearn.feature_extraction.DictVectorizer`.
+#
+# **FeatureHasher on raw tokens**
+#
+# Alternatively, one can set `input_type="string"` in the
+# :func:`~sklearn.feature_extraction.FeatureHasher` to vectorize the strings
+# output directly from the customized `tokenize` function. This is equivalent to
+# passing a dictionary with an implied frequency of 1 for each feature name.
+
+t0 = time()
+hasher = FeatureHasher(n_features=2**18, input_type="string")
+X = hasher.transform(tokenize(d) for d in raw_data)
+duration = time() - t0
+dict_count_vectorizers["vectorizer"].append(
+ hasher.__class__.__name__ + "\non raw tokens"
+)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {n_nonzero_columns(X)} unique tokens")
+
+# %%
+# We now plot the speed of the above methods for vectorizing.
+
+import matplotlib.pyplot as plt
+
+fig, ax = plt.subplots(figsize=(12, 6))
+
+y_pos = np.arange(len(dict_count_vectorizers["vectorizer"]))
+ax.barh(y_pos, dict_count_vectorizers["speed"], align="center")
+ax.set_yticks(y_pos)
+ax.set_yticklabels(dict_count_vectorizers["vectorizer"])
+ax.invert_yaxis()
+_ = ax.set_xlabel("speed (MB/s)")
+
+# %%
+# In both cases :func:`~sklearn.feature_extraction.FeatureHasher` is
+# approximately twice as fast as
+# :func:`~sklearn.feature_extraction.DictVectorizer`. This is handy when dealing
+# with large amounts of data, with the downside of losing the invertibility of
+# the transformation, which in turn makes the interpretation of a model a more
+# complex task.
+#
+# The `FeatureHeasher` with `input_type="string"` is slightly faster than the
+# variant that works on frequency dict because it does not count repeated
+# tokens: each token is implicitly counted once, even if it was repeated.
+# Depending on the downstream machine learning task, it can be a limitation or
+# not.
+#
+# Comparison with special purpose text vectorizers
+# ------------------------------------------------
+#
+# :func:`~sklearn.feature_extraction.text.CountVectorizer` accepts raw data as
+# it internally implements tokenization and occurrence counting. It is similar
+# to the :func:`~sklearn.feature_extraction.DictVectorizer` when used along with
+# the customized function `token_freqs` as done in the previous section. The
+# difference being that :func:`~sklearn.feature_extraction.text.CountVectorizer`
+# is more flexible. In particular it accepts various regex patterns through the
+# `token_pattern` parameter.
+
+from sklearn.feature_extraction.text import CountVectorizer
+
+t0 = time()
+vectorizer = CountVectorizer()
+vectorizer.fit_transform(raw_data)
+duration = time() - t0
+dict_count_vectorizers["vectorizer"].append(vectorizer.__class__.__name__)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {len(vectorizer.get_feature_names_out())} unique terms")
+
+# %%
+# We see that using the :func:`~sklearn.feature_extraction.text.CountVectorizer`
+# implementation is approximately twice as fast as using the
+# :func:`~sklearn.feature_extraction.DictVectorizer` along with the simple
+# function we defined for mapping the tokens. The reason is that
+# :func:`~sklearn.feature_extraction.text.CountVectorizer` is optimized by
+# reusing a compiled regular expression for the full training set instead of
+# creating one per document as done in our naive tokenize function.
+#
+# Now we make a similar experiment with the
+# :func:`~sklearn.feature_extraction.text.HashingVectorizer`, which is
+# equivalent to combining the “hashing trick” implemented by the
+# :func:`~sklearn.feature_extraction.FeatureHasher` class and the text
+# preprocessing and tokenization of the
+# :func:`~sklearn.feature_extraction.text.CountVectorizer`.
+
+from sklearn.feature_extraction.text import HashingVectorizer
+
+t0 = time()
+vectorizer = HashingVectorizer(n_features=2**18)
+vectorizer.fit_transform(raw_data)
+duration = time() - t0
+dict_count_vectorizers["vectorizer"].append(vectorizer.__class__.__name__)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+
+# %%
+# We can observe that this is the fastest text tokenization strategy so far,
+# assuming that the downstream machine learning task can tolerate a few
+# collisions.
+#
+# TfidfVectorizer
+# ---------------
+#
+# In a large text corpus, some words appear with higher frequency (e.g. “the”,
+# “a”, “is” in English) and do not carry meaningful information about the actual
+# contents of a document. If we were to feed the word count data directly to a
+# classifier, those very common terms would shadow the frequencies of rarer yet
+# more informative terms. In order to re-weight the count features into floating
+# point values suitable for usage by a classifier it is very common to use the
+# tf–idf transform as implemented by the
+# :func:`~sklearn.feature_extraction.text.TfidfTransformer`. TF stands for
+# "term-frequency" while "tf–idf" means term-frequency times inverse
+# document-frequency.
+#
+# We now benchmark the :func:`~sklearn.feature_extraction.text.TfidfVectorizer`,
+# which is equivalent to combining the tokenization and occurrence counting of
+# the :func:`~sklearn.feature_extraction.text.CountVectorizer` along with the
+# normalizing and weighting from a
+# :func:`~sklearn.feature_extraction.text.TfidfTransformer`.
+
+from sklearn.feature_extraction.text import TfidfVectorizer
+
+t0 = time()
+vectorizer = TfidfVectorizer()
+vectorizer.fit_transform(raw_data)
+duration = time() - t0
+dict_count_vectorizers["vectorizer"].append(vectorizer.__class__.__name__)
+dict_count_vectorizers["speed"].append(data_size_mb / duration)
+print(f"done in {duration:.3f} s at {data_size_mb / duration:.1f} MB/s")
+print(f"Found {len(vectorizer.get_feature_names_out())} unique terms")
+
+# %%
+# Summary
+# -------
+# Let's conclude this notebook by summarizing all the recorded processing speeds
+# in a single plot:
+
+fig, ax = plt.subplots(figsize=(12, 6))
+
+y_pos = np.arange(len(dict_count_vectorizers["vectorizer"]))
+ax.barh(y_pos, dict_count_vectorizers["speed"], align="center")
+ax.set_yticks(y_pos)
+ax.set_yticklabels(dict_count_vectorizers["vectorizer"])
+ax.invert_yaxis()
+_ = ax.set_xlabel("speed (MB/s)")
+
+# %%
+# Notice from the plot that
+# :func:`~sklearn.feature_extraction.text.TfidfVectorizer` is slightly slower
+# than :func:`~sklearn.feature_extraction.text.CountVectorizer` because of the
+# extra operation induced by the
+# :func:`~sklearn.feature_extraction.text.TfidfTransformer`.
+#
+# Also notice that, by setting the number of features `n_features = 2**18`, the
+# :func:`~sklearn.feature_extraction.text.HashingVectorizer` performs better
+# than the :func:`~sklearn.feature_extraction.text.CountVectorizer` at the
+# expense of inversibility of the transformation due to hash collisions.
+#
+# We highlight that :func:`~sklearn.feature_extraction.text.CountVectorizer` and
+# :func:`~sklearn.feature_extraction.text.HashingVectorizer` perform better than
+# their equivalent :func:`~sklearn.feature_extraction.DictVectorizer` and
+# :func:`~sklearn.feature_extraction.FeatureHasher` on manually tokenized
+# documents since the internal tokenization step of the former vectorizers
+# compiles a regular expression once and then reuses it for all the documents.
diff --git a/lgtm.yml b/lgtm.yml
index 5394cc1664bc9..2dfdba3d160cc 100644
--- a/lgtm.yml
+++ b/lgtm.yml
@@ -1,7 +1,7 @@
extraction:
cpp:
before_index:
- - pip3 install numpy==1.16.3
+ - pip3 install numpy==1.17.3
- pip3 install --no-deps scipy Cython
index:
build_command:
diff --git a/maint_tools/update_tracking_issue.py b/maint_tools/update_tracking_issue.py
index 855c733cffb31..4ddc9d1bfe8e6 100644
--- a/maint_tools/update_tracking_issue.py
+++ b/maint_tools/update_tracking_issue.py
@@ -14,6 +14,7 @@
from pathlib import Path
import sys
import argparse
+from datetime import datetime, timezone
import defusedxml.ElementTree as ET
from github import Github
@@ -56,6 +57,8 @@
gh = Github(args.bot_github_token)
issue_repo = gh.get_repo(args.issue_repo)
+dt_now = datetime.now(tz=timezone.utc)
+date_str = dt_now.strftime("%b %d, %Y")
title = f"⚠️ CI failed on {args.ci_name} ⚠️"
@@ -71,18 +74,28 @@ def get_issue():
def create_or_update_issue(body=""):
# Interact with GitHub API to create issue
- header = f"**CI Failed on [{args.ci_name}]({args.link_to_ci_run})**"
- body_text = f"{header}\n{body}"
+ link = f"[{args.ci_name}]({args.link_to_ci_run})"
issue = get_issue()
+ max_body_length = 60_000
+ original_body_length = len(body)
+ # Avoid "body is too long (maximum is 65536 characters)" error from github REST API
+ if original_body_length > max_body_length:
+ body = (
+ f"{body[:max_body_length]}\n...\n"
+ f"Body was too long ({original_body_length} characters) and was shortened"
+ )
+
if issue is None:
# Create new issue
- issue = issue_repo.create_issue(title=title, body=body_text)
+ header = f"**CI failed on {link}** ({date_str})"
+ issue = issue_repo.create_issue(title=title, body=f"{header}\n{body}")
print(f"Created issue in {args.issue_repo}#{issue.number}")
sys.exit()
else:
- # Add comment to existing issue
- issue.create_comment(body=body_text)
+ # Update existing issue
+ header = f"**CI is still failing on {link}** ({date_str})"
+ issue.edit(body=f"{header}\n{body}")
print(f"Commented on issue: {args.issue_repo}#{issue.number}")
sys.exit()
@@ -91,11 +104,21 @@ def close_issue_if_opened():
print("Test has no failures!")
issue = get_issue()
if issue is not None:
- comment = (
- f"## CI is no longer failing! ✅\n\n[Successful run]({args.link_to_ci_run})"
+ header_str = "## CI is no longer failing!"
+ comment_str = (
+ f"{header_str} ✅\n\n[Successful run]({args.link_to_ci_run}) on {date_str}"
)
+
print(f"Commented on issue #{issue.number}")
- issue.create_comment(body=comment)
+ # New comment if "## CI is no longer failing!" comment does not exist
+ # If it does exist update the original comment which includes the new date
+ for comment in issue.get_comments():
+ if comment.body.startswith(header_str):
+ comment.edit(body=comment_str)
+ break
+ else: # no break
+ issue.create_comment(body=comment_str)
+
if args.auto_close.lower() == "true":
print(f"Closing issue #{issue.number}")
issue.edit(state="closed")
diff --git a/setup.cfg b/setup.cfg
index 2d59f866547c8..d5c4bba3e5a86 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -33,7 +33,7 @@ ignore=
E226, # missing whitespace around arithmetic operator
E704, # multiple statements on one line (def)
E731, # do not assign a lambda expression, use a def
- E741, # do not use variables named ‘l’, ‘O’, or ‘I’
+ E741, # do not use variables named 'l', 'O', or 'I'
W503, # line break before binary operator
W504 # line break after binary operator
exclude=
diff --git a/setup.py b/setup.py
index 7ad32e95e53a5..2ecc5ba0bcc2e 100755
--- a/setup.py
+++ b/setup.py
@@ -254,7 +254,7 @@ def setup_package():
classifiers=[
"Intended Audience :: Science/Research",
"Intended Audience :: Developers",
- "License :: OSI Approved",
+ "License :: OSI Approved :: BSD License",
"Programming Language :: C",
"Programming Language :: Python",
"Topic :: Software Development",
@@ -290,6 +290,7 @@ def setup_package():
from setuptools import setup
metadata["version"] = VERSION
+ metadata["packages"] = ["sklearn"]
else:
if sys.version_info < required_python_version:
required_version = "%d.%d" % required_python_version
diff --git a/sklearn/__init__.py b/sklearn/__init__.py
index c11158380d3c3..4e335ed928904 100644
--- a/sklearn/__init__.py
+++ b/sklearn/__init__.py
@@ -39,7 +39,7 @@
# Dev branch marker is: 'X.Y.dev' or 'X.Y.devN' where N is an integer.
# 'X.Y.dev0' is the canonical version of 'X.Y.dev'
#
-__version__ = "1.1.1"
+__version__ = "1.1.2"
# On OSX, we can get a runtime error due to multiple OpenMP libraries loaded
diff --git a/sklearn/_min_dependencies.py b/sklearn/_min_dependencies.py
index 957e1e01f0551..3e28d6bc7dc98 100644
--- a/sklearn/_min_dependencies.py
+++ b/sklearn/_min_dependencies.py
@@ -1,4 +1,5 @@
"""All minimum dependencies for scikit-learn."""
+from collections import defaultdict
import platform
import argparse
@@ -30,7 +31,7 @@
"threadpoolctl": (THREADPOOLCTL_MIN_VERSION, "install"),
"cython": (CYTHON_MIN_VERSION, "build"),
"matplotlib": ("3.1.2", "benchmark, docs, examples, tests"),
- "scikit-image": ("0.14.5", "docs, examples, tests"),
+ "scikit-image": ("0.16.2", "docs, examples, tests"),
"pandas": ("1.0.5", "benchmark, docs, examples, tests"),
"seaborn": ("0.9.0", "docs, examples"),
"memory_profiler": ("0.57.0", "benchmark, docs"),
@@ -38,7 +39,7 @@
"pytest-cov": ("2.9.0", "tests"),
"flake8": ("3.8.2", "tests"),
"black": ("22.3.0", "tests"),
- "mypy": ("0.770", "tests"),
+ "mypy": ("0.961", "tests"),
"pyamg": ("4.0.0", "tests"),
"sphinx": ("4.0.1", "docs"),
"sphinx-gallery": ("0.7.0", "docs"),
@@ -46,14 +47,14 @@
"Pillow": ("7.1.2", "docs"),
"sphinx-prompt": ("1.3.0", "docs"),
"sphinxext-opengraph": ("0.4.2", "docs"),
+ # XXX: Pin conda-lock to the latest released version (needs manual update
+ # from time to time)
+ "conda-lock": ("1.0.5", "maintenance"),
}
# create inverse mapping for setuptools
-tag_to_packages: dict = {
- extra: []
- for extra in ["build", "install", "docs", "examples", "tests", "benchmark"]
-}
+tag_to_packages: dict = defaultdict(list)
for package, (min_version, extras) in dependent_packages.items():
for extra in extras.split(", "):
tag_to_packages[extra].append("{}>={}".format(package, min_version))
diff --git a/sklearn/base.py b/sklearn/base.py
index 3bf19fb96f0c3..8e64ab3ac4156 100644
--- a/sklearn/base.py
+++ b/sklearn/base.py
@@ -209,7 +209,7 @@ def get_params(self, deep=True):
out = dict()
for key in self._get_param_names():
value = getattr(self, key)
- if deep and hasattr(value, "get_params"):
+ if deep and hasattr(value, "get_params") and not isinstance(value, type):
deep_items = value.get_params().items()
out.update((key + "__" + k, val) for k, val in deep_items)
out[key] = value
diff --git a/sklearn/cluster/_bicluster.py b/sklearn/cluster/_bicluster.py
index a360802009f2c..c8b46fa38c6d1 100644
--- a/sklearn/cluster/_bicluster.py
+++ b/sklearn/cluster/_bicluster.py
@@ -385,7 +385,7 @@ class SpectralBiclustering(BaseSpectral):
default is 'bistochastic'.
.. warning::
- if `method='log'`, the data must be sparse.
+ if `method='log'`, the data must not be sparse.
n_components : int, default=6
Number of singular vectors to check.
diff --git a/sklearn/cluster/_birch.py b/sklearn/cluster/_birch.py
index cfdfeab27b15c..604365a184af7 100644
--- a/sklearn/cluster/_birch.py
+++ b/sklearn/cluster/_birch.py
@@ -89,6 +89,11 @@ def _split_node(node, threshold, branching_factor):
node1_dist, node2_dist = dist[(farthest_idx,)]
node1_closer = node1_dist < node2_dist
+ # make sure node1 is closest to itself even if all distances are equal.
+ # This can only happen when all node.centroids_ are duplicates leading to all
+ # distances between centroids being zero.
+ node1_closer[farthest_idx[0]] = True
+
for idx, subcluster in enumerate(node.subclusters_):
if node1_closer[idx]:
new_node1.append_subcluster(subcluster)
diff --git a/sklearn/cluster/_bisect_k_means.py b/sklearn/cluster/_bisect_k_means.py
index c7dc2c5a772e5..90ca5107f67b4 100644
--- a/sklearn/cluster/_bisect_k_means.py
+++ b/sklearn/cluster/_bisect_k_means.py
@@ -183,7 +183,7 @@ class BisectingKMeans(_BaseKMeans):
Notes
-----
- It might be inefficient when n_cluster is less than 3, due to unnecassary
+ It might be inefficient when n_cluster is less than 3, due to unnecessary
calculations for that case.
Examples
diff --git a/sklearn/cluster/_kmeans.py b/sklearn/cluster/_kmeans.py
index eca8a5c2dc3ce..70a595936b41c 100644
--- a/sklearn/cluster/_kmeans.py
+++ b/sklearn/cluster/_kmeans.py
@@ -642,7 +642,7 @@ def _kmeans_single_lloyd(
strict_convergence = False
# Threadpoolctl context to limit the number of threads in second level of
- # nested parallelism (i.e. BLAS) to avoid oversubsciption.
+ # nested parallelism (i.e. BLAS) to avoid oversubscription.
with threadpool_limits(limits=1, user_api="blas"):
for i in range(max_iter):
lloyd_iter(
@@ -1139,9 +1139,11 @@ class KMeans(_BaseKMeans):
(n_clusters, n_features), default='k-means++'
Method for initialization:
- 'k-means++' : selects initial cluster centers for k-mean
- clustering in a smart way to speed up convergence. See section
- Notes in k_init for more details.
+ 'k-means++' : selects initial cluster centroids using sampling based on
+ an empirical probability distribution of the points' contribution to the
+ overall inertia. This technique speeds up convergence, and is
+ theoretically proven to be :math:`\\mathcal{O}(\\log k)`-optimal.
+ See the description of `n_init` for more details.
'random': choose `n_clusters` observations (rows) at random from data
for the initial centroids.
@@ -1601,9 +1603,11 @@ class MiniBatchKMeans(_BaseKMeans):
(n_clusters, n_features), default='k-means++'
Method for initialization:
- 'k-means++' : selects initial cluster centers for k-mean
- clustering in a smart way to speed up convergence. See section
- Notes in k_init for more details.
+ 'k-means++' : selects initial cluster centroids using sampling based on
+ an empirical probability distribution of the points' contribution to the
+ overall inertia. This technique speeds up convergence, and is
+ theoretically proven to be :math:`\\mathcal{O}(\\log k)`-optimal.
+ See the description of `n_init` for more details.
'random': choose `n_clusters` observations (rows) at random from data
for the initial centroids.
@@ -1667,8 +1671,10 @@ class MiniBatchKMeans(_BaseKMeans):
n_init : int, default=3
Number of random initializations that are tried.
- In contrast to KMeans, the algorithm is only run once, using the
- best of the ``n_init`` initializations as measured by inertia.
+ In contrast to KMeans, the algorithm is only run once, using the best of
+ the `n_init` initializations as measured by inertia. Several runs are
+ recommended for sparse high-dimensional problems (see
+ :ref:`kmeans_sparse_high_dim`).
reassignment_ratio : float, default=0.01
Control the fraction of the maximum number of counts for a center to
@@ -1724,6 +1730,12 @@ class MiniBatchKMeans(_BaseKMeans):
-----
See https://www.eecs.tufts.edu/~dsculley/papers/fastkmeans.pdf
+ When there are too few points in the dataset, some centers may be
+ duplicated, which means that a proper clustering in terms of the number
+ of requesting clusters and the number of returned clusters will not
+ always match. One solution is to set `reassignment_ratio=0`, which
+ prevents reassignments of clusters that are too small.
+
Examples
--------
>>> from sklearn.cluster import MiniBatchKMeans
diff --git a/sklearn/cluster/_spectral.py b/sklearn/cluster/_spectral.py
index 390b567c0d0bb..22bf204f2094c 100644
--- a/sklearn/cluster/_spectral.py
+++ b/sklearn/cluster/_spectral.py
@@ -561,11 +561,11 @@ class SpectralClustering(ClusterMixin, BaseEstimator):
Stella X. Yu, Jianbo Shi
`_
- .. [4] `Toward the Optimal Preconditioned Eigensolver:
- Locally Optimal Block Preconditioned Conjugate Gradient Method, 2001.
+ .. [4] :doi:`Toward the Optimal Preconditioned Eigensolver:
+ Locally Optimal Block Preconditioned Conjugate Gradient Method, 2001
A. V. Knyazev
SIAM Journal on Scientific Computing 23, no. 2, pp. 517-541.
- `_
+ <10.1137/S1064827500366124>`
.. [5] :doi:`Simple, direct, and efficient multi-way spectral clustering, 2019
Anil Damle, Victor Minden, Lexing Ying
diff --git a/sklearn/cluster/tests/test_birch.py b/sklearn/cluster/tests/test_birch.py
index e0051704653ae..a42f421c8f248 100644
--- a/sklearn/cluster/tests/test_birch.py
+++ b/sklearn/cluster/tests/test_birch.py
@@ -228,3 +228,39 @@ def test_feature_names_out():
names_out = brc.get_feature_names_out()
assert_array_equal([f"birch{i}" for i in range(n_clusters)], names_out)
+
+
+def test_both_subclusters_updated():
+ """Check that both subclusters are updated when a node a split, even when there are
+ duplicated data points. Non-regression test for #23269.
+ """
+
+ X = np.array(
+ [
+ [-2.6192791, -1.5053215],
+ [-2.9993038, -1.6863596],
+ [-2.3724914, -1.3438171],
+ [-2.336792, -1.3417323],
+ [-2.4089134, -1.3290224],
+ [-2.3724914, -1.3438171],
+ [-3.364009, -1.8846745],
+ [-2.3724914, -1.3438171],
+ [-2.617677, -1.5003285],
+ [-2.2960556, -1.3260119],
+ [-2.3724914, -1.3438171],
+ [-2.5459878, -1.4533926],
+ [-2.25979, -1.3003055],
+ [-2.4089134, -1.3290224],
+ [-2.3724914, -1.3438171],
+ [-2.4089134, -1.3290224],
+ [-2.5459878, -1.4533926],
+ [-2.3724914, -1.3438171],
+ [-2.9720619, -1.7058647],
+ [-2.336792, -1.3417323],
+ [-2.3724914, -1.3438171],
+ ],
+ dtype=np.float32,
+ )
+
+ # no error
+ Birch(branching_factor=5, threshold=1e-5, n_clusters=None).fit(X)
diff --git a/sklearn/compose/_column_transformer.py b/sklearn/compose/_column_transformer.py
index 15f1424498856..84e122fab3cc8 100644
--- a/sklearn/compose/_column_transformer.py
+++ b/sklearn/compose/_column_transformer.py
@@ -190,6 +190,23 @@ class ColumnTransformer(TransformerMixin, _BaseComposition):
>>> ct.fit_transform(X)
array([[0. , 1. , 0.5, 0.5],
[0.5, 0.5, 0. , 1. ]])
+
+ :class:`ColumnTransformer` can be configured with a transformer that requires
+ a 1d array by setting the column to a string:
+
+ >>> from sklearn.feature_extraction import FeatureHasher
+ >>> from sklearn.preprocessing import MinMaxScaler
+ >>> import pandas as pd # doctest: +SKIP
+ >>> X = pd.DataFrame({
+ ... "documents": ["First item", "second one here", "Is this the last?"],
+ ... "width": [3, 4, 5],
+ ... }) # doctest: +SKIP
+ >>> # "documents" is a string which configures ColumnTransformer to
+ >>> # pass the documents column as a 1d array to the FeatureHasher
+ >>> ct = ColumnTransformer(
+ ... [("text_preprocess", FeatureHasher(input_type="string"), "documents"),
+ ... ("num_preprocess", MinMaxScaler(), ["width"])])
+ >>> X_trans = ct.fit_transform(X) # doctest: +SKIP
"""
_required_parameters = ["transformers"]
diff --git a/sklearn/covariance/_empirical_covariance.py b/sklearn/covariance/_empirical_covariance.py
index 4362a14f04f6e..532390d74bded 100644
--- a/sklearn/covariance/_empirical_covariance.py
+++ b/sklearn/covariance/_empirical_covariance.py
@@ -207,7 +207,7 @@ def get_precision(self):
return precision
def fit(self, X, y=None):
- """Fit the maximum liklihood covariance estimator to X.
+ """Fit the maximum likelihood covariance estimator to X.
Parameters
----------
diff --git a/sklearn/cross_decomposition/_pls.py b/sklearn/cross_decomposition/_pls.py
index 8a804142e13bb..905e0d108d40b 100644
--- a/sklearn/cross_decomposition/_pls.py
+++ b/sklearn/cross_decomposition/_pls.py
@@ -170,7 +170,7 @@ class _PLS(
Main ref: Wegelin, a survey of Partial Least Squares (PLS) methods,
with emphasis on the two-block case
- https://www.stat.washington.edu/research/reports/2000/tr371.pdf
+ https://stat.uw.edu/sites/default/files/files/reports/2000/tr371.pdf
"""
@abstractmethod
@@ -1038,7 +1038,7 @@ def transform(self, X, Y=None):
Returns
-------
x_scores : array-like or tuple of array-like
- The transformed data `X_tranformed` if `Y is not None`,
+ The transformed data `X_transformed` if `Y is not None`,
`(X_transformed, Y_transformed)` otherwise.
"""
check_is_fitted(self)
@@ -1069,7 +1069,7 @@ def fit_transform(self, X, y=None):
Returns
-------
out : array-like or tuple of array-like
- The transformed data `X_tranformed` if `Y is not None`,
+ The transformed data `X_transformed` if `Y is not None`,
`(X_transformed, Y_transformed)` otherwise.
"""
return self.fit(X, y).transform(X, y)
diff --git a/sklearn/datasets/_base.py b/sklearn/datasets/_base.py
index 367816fa4a467..a9a5f3b39c3ec 100644
--- a/sklearn/datasets/_base.py
+++ b/sklearn/datasets/_base.py
@@ -346,7 +346,7 @@ def load_gzip_compressed_csv_data(
encoding="utf-8",
**kwargs,
):
- """Loads gzip-compressed `data_file_name` from `data_module` with `importlib.resources`.
+ """Loads gzip-compressed with `importlib.resources`.
1) Open resource file with `importlib.resources.open_binary`
2) Decompress file obj with `gzip.open`
diff --git a/sklearn/datasets/_kddcup99.py b/sklearn/datasets/_kddcup99.py
index b698d299b7c8d..f9d609d362c04 100644
--- a/sklearn/datasets/_kddcup99.py
+++ b/sklearn/datasets/_kddcup99.py
@@ -133,6 +133,10 @@ def fetch_kddcup99(
The names of the target columns
(data, target) : tuple if ``return_X_y`` is True
+ A tuple of two ndarray. The first containing a 2D array of
+ shape (n_samples, n_features) with each row representing one
+ sample and each column representing the features. The second
+ ndarray of shape (n_samples,) containing the target samples.
.. versionadded:: 0.20
"""
@@ -225,7 +229,6 @@ def fetch_kddcup99(
def _fetch_brute_kddcup99(data_home=None, download_if_missing=True, percent10=True):
-
"""Load the kddcup99 dataset, downloading it if necessary.
Parameters
diff --git a/sklearn/datasets/_lfw.py b/sklearn/datasets/_lfw.py
index dc1267af59f96..33c1234f907b7 100644
--- a/sklearn/datasets/_lfw.py
+++ b/sklearn/datasets/_lfw.py
@@ -308,13 +308,15 @@ def fetch_lfw_people(
target : numpy array of shape (13233,)
Labels associated to each face image.
Those labels range from 0-5748 and correspond to the person IDs.
+ target_names : numpy array of shape (5749,)
+ Names of all persons in the dataset.
+ Position in array corresponds to the person ID in the target array.
DESCR : str
Description of the Labeled Faces in the Wild (LFW) dataset.
(data, target) : tuple if ``return_X_y`` is True
.. versionadded:: 0.20
-
"""
lfw_home, data_folder_path = _check_fetch_lfw(
data_home=data_home, funneled=funneled, download_if_missing=download_if_missing
@@ -464,7 +466,7 @@ def fetch_lfw_pairs(
slice_ : tuple of slice, default=(slice(70, 195), slice(78, 172))
Provide a custom 2D slice (height, width) to extract the
'interesting' part of the jpeg files and avoid use statistical
- correlation from the background
+ correlation from the background.
download_if_missing : bool, default=True
If False, raise a IOError if the data is not locally available
@@ -489,9 +491,11 @@ def fetch_lfw_pairs(
target : numpy array of shape (2200,). Shape depends on ``subset``.
Labels associated to each pair of images.
The two label values being different persons or the same person.
+ target_names : numpy array of shape (2,)
+ Explains the target values of the target array.
+ 0 corresponds to "Different person", 1 corresponds to "same person".
DESCR : str
Description of the Labeled Faces in the Wild (LFW) dataset.
-
"""
lfw_home, data_folder_path = _check_fetch_lfw(
data_home=data_home, funneled=funneled, download_if_missing=download_if_missing
diff --git a/sklearn/datasets/_samples_generator.py b/sklearn/datasets/_samples_generator.py
index 4bb61cf85dbca..c144f9b255a06 100644
--- a/sklearn/datasets/_samples_generator.py
+++ b/sklearn/datasets/_samples_generator.py
@@ -1405,7 +1405,7 @@ def make_spd_matrix(n_dim, *, random_state=None):
See Also
--------
- make_sparse_spd_matrix
+ make_sparse_spd_matrix: Generate a sparse symmetric definite positive matrix.
"""
generator = check_random_state(random_state)
@@ -1532,9 +1532,9 @@ def make_swiss_roll(n_samples=100, *, noise=0.0, random_state=None, hole=False):
References
----------
- .. [1] S. Marsland, "Machine Learning: An Algorithmic Perspective",
- Chapter 10, 2009.
- http://seat.massey.ac.nz/personal/s.r.marsland/Code/10/lle.py
+ .. [1] S. Marsland, "Machine Learning: An Algorithmic Perspective", 2nd edition,
+ Chapter 6, 2014.
+ https://homepages.ecs.vuw.ac.nz/~marslast/Code/Ch6/lle.py
"""
generator = check_random_state(random_state)
@@ -1637,7 +1637,7 @@ def make_gaussian_quantiles(
The number of features for each sample.
n_classes : int, default=3
- The number of classes
+ The number of classes.
shuffle : bool, default=True
Shuffle the samples.
@@ -1662,7 +1662,6 @@ def make_gaussian_quantiles(
References
----------
.. [1] J. Zhu, H. Zou, S. Rosset, T. Hastie, "Multi-class AdaBoost", 2009.
-
"""
if n_samples < n_classes:
raise ValueError("n_samples must be at least n_classes")
diff --git a/sklearn/datasets/descr/olivetti_faces.rst b/sklearn/datasets/descr/olivetti_faces.rst
index c6193d5056538..4feadcc4b2fb1 100644
--- a/sklearn/datasets/descr/olivetti_faces.rst
+++ b/sklearn/datasets/descr/olivetti_faces.rst
@@ -9,7 +9,7 @@ April 1994 at AT&T Laboratories Cambridge. The
fetching / caching function that downloads the data
archive from AT&T.
-.. _This dataset contains a set of face images: http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
+.. _This dataset contains a set of face images: https://cam-orl.co.uk/facedatabase.html
As described on the original website:
diff --git a/sklearn/datasets/descr/twenty_newsgroups.rst b/sklearn/datasets/descr/twenty_newsgroups.rst
index a7542ea57d529..3a327a4cbc19c 100644
--- a/sklearn/datasets/descr/twenty_newsgroups.rst
+++ b/sklearn/datasets/descr/twenty_newsgroups.rst
@@ -226,7 +226,7 @@ the ``--filter`` option to compare the results.
discussion sparked by the death of George Floyd and a national reckoning over
race and colonialism, the Cleveland Indians have decided to change their
name." Team owner Paul Dolan "did make it clear that the team will not make
- its informal nickname -- the Tribe -- its new team name." "It’s not going to
+ its informal nickname -- the Tribe -- its new team name." "It's not going to
be a half-step away from the Indians," Dolan said."We will not have a Native
American-themed name."
diff --git a/sklearn/datasets/descr/wine_data.rst b/sklearn/datasets/descr/wine_data.rst
index bfde9288fa4dd..dbe7f38e44aa6 100644
--- a/sklearn/datasets/descr/wine_data.rst
+++ b/sklearn/datasets/descr/wine_data.rst
@@ -5,7 +5,7 @@ Wine recognition dataset
**Data Set Characteristics:**
- :Number of Instances: 178 (50 in each of three classes)
+ :Number of Instances: 178
:Number of Attributes: 13 numeric, predictive attributes and the class
:Attribute Information:
- Alcohol
diff --git a/sklearn/decomposition/_factor_analysis.py b/sklearn/decomposition/_factor_analysis.py
index 4b8eab3492ca8..303e7ea280b93 100644
--- a/sklearn/decomposition/_factor_analysis.py
+++ b/sklearn/decomposition/_factor_analysis.py
@@ -45,7 +45,7 @@ class FactorAnalysis(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEst
If we would restrict the model further, by assuming that the Gaussian
noise is even isotropic (all diagonal entries are the same) we would obtain
- :class:`PPCA`.
+ :class:`PCA`.
FactorAnalysis performs a maximum likelihood estimate of the so-called
`loading` matrix, the transformation of the latent variables to the
diff --git a/sklearn/decomposition/_fastica.py b/sklearn/decomposition/_fastica.py
index 490a3323344d1..a6ddac4cb3347 100644
--- a/sklearn/decomposition/_fastica.py
+++ b/sklearn/decomposition/_fastica.py
@@ -180,22 +180,23 @@ def fastica(
`n_features` is the number of features.
n_components : int, default=None
- Number of components to extract. If None no dimension reduction
- is performed.
+ Number of components to use. If None is passed, all are used.
algorithm : {'parallel', 'deflation'}, default='parallel'
- Apply a parallel or deflational FASTICA algorithm.
+ Specify which algorithm to use for FastICA.
whiten : str or bool, default="warn"
Specify the whitening strategy to use.
- If 'arbitrary-variance' (default), a whitening with variance arbitrary is used.
- If 'unit-variance', the whitening matrix is rescaled to ensure that each
- recovered source has unit variance.
- If False, the data is already considered to be whitened, and no
- whitening is performed.
+
+ - If 'arbitrary-variance' (default), a whitening with variance
+ arbitrary is used.
+ - If 'unit-variance', the whitening matrix is rescaled to ensure that
+ each recovered source has unit variance.
+ - If False, the data is already considered to be whitened, and no
+ whitening is performed.
.. deprecated:: 1.1
- From version 1.3, `whiten='unit-variance'` will be used by default.
+ Starting in v1.3, `whiten='unit-variance'` will be used by default.
`whiten=True` is deprecated from 1.1 and will raise ValueError in 1.3.
Use `whiten=arbitrary-variance` instead.
@@ -206,10 +207,10 @@ def fastica(
You can also provide your own function. It should return a tuple
containing the value of the function, and of its derivative, in the
point. The derivative should be averaged along its last dimension.
- Example:
+ Example::
- def my_g(x):
- return x ** 3, np.mean(3 * x ** 2, axis=-1)
+ def my_g(x):
+ return x ** 3, (3 * x ** 2).mean(axis=-1)
fun_args : dict, default=None
Arguments to send to the functional form.
@@ -219,13 +220,13 @@ def my_g(x):
max_iter : int, default=200
Maximum number of iterations to perform.
- tol : float, default=1e-04
+ tol : float, default=1e-4
A positive scalar giving the tolerance at which the
un-mixing matrix is considered to have converged.
w_init : ndarray of shape (n_components, n_components), default=None
- Initial un-mixing array of dimension (n.comp,n.comp).
- If None (default) then an array of normal r.v.'s is used.
+ Initial un-mixing array. If `w_init=None`, then an array of values
+ drawn from a normal distribution is used.
random_state : int, RandomState instance or None, default=None
Used to initialize ``w_init`` when not specified, with a
@@ -332,18 +333,20 @@ class FastICA(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstimator)
Number of components to use. If None is passed, all are used.
algorithm : {'parallel', 'deflation'}, default='parallel'
- Apply parallel or deflational algorithm for FastICA.
+ Specify which algorithm to use for FastICA.
whiten : str or bool, default="warn"
Specify the whitening strategy to use.
- If 'arbitrary-variance' (default), a whitening with variance arbitrary is used.
- If 'unit-variance', the whitening matrix is rescaled to ensure that each
- recovered source has unit variance.
- If False, the data is already considered to be whitened, and no
- whitening is performed.
+
+ - If 'arbitrary-variance' (default), a whitening with variance
+ arbitrary is used.
+ - If 'unit-variance', the whitening matrix is rescaled to ensure that
+ each recovered source has unit variance.
+ - If False, the data is already considered to be whitened, and no
+ whitening is performed.
.. deprecated:: 1.1
- From version 1.3 whiten='unit-variance' will be used by default.
+ Starting in v1.3, `whiten='unit-variance'` will be used by default.
`whiten=True` is deprecated from 1.1 and will raise ValueError in 1.3.
Use `whiten=arbitrary-variance` instead.
@@ -353,24 +356,27 @@ class FastICA(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstimator)
or 'cube'.
You can also provide your own function. It should return a tuple
containing the value of the function, and of its derivative, in the
- point. Example::
+ point. The derivative should be averaged along its last dimension.
+ Example::
def my_g(x):
return x ** 3, (3 * x ** 2).mean(axis=-1)
fun_args : dict, default=None
Arguments to send to the functional form.
- If empty and if fun='logcosh', fun_args will take value
+ If empty or None and if fun='logcosh', fun_args will take value
{'alpha' : 1.0}.
max_iter : int, default=200
Maximum number of iterations during fit.
tol : float, default=1e-4
- Tolerance on update at each iteration.
+ A positive scalar giving the tolerance at which the
+ un-mixing matrix is considered to have converged.
w_init : ndarray of shape (n_components, n_components), default=None
- The mixing matrix to be used to initialize the algorithm.
+ Initial un-mixing array. If `w_init=None`, then an array of values
+ drawn from a normal distribution is used.
random_state : int, RandomState instance or None, default=None
Used to initialize ``w_init`` when not specified, with a
@@ -486,14 +492,14 @@ def _fit(self, X, compute_sources=False):
if self._whiten == "warn":
warnings.warn(
- "From version 1.3 whiten='unit-variance' will be used by default.",
+ "Starting in v1.3, whiten='unit-variance' will be used by default.",
FutureWarning,
)
self._whiten = "arbitrary-variance"
if self._whiten is True:
warnings.warn(
- "From version 1.3 whiten=True should be specified as "
+ "Starting in v1.3, whiten=True should be specified as "
"whiten='arbitrary-variance' (its current behaviour). This "
"behavior is deprecated in 1.1 and will raise ValueError in 1.3.",
FutureWarning,
diff --git a/sklearn/decomposition/_incremental_pca.py b/sklearn/decomposition/_incremental_pca.py
index 589796a7c97f7..da050a7e42efa 100644
--- a/sklearn/decomposition/_incremental_pca.py
+++ b/sklearn/decomposition/_incremental_pca.py
@@ -137,10 +137,9 @@ class IncrementalPCA(_BasePCA):
See https://www.cs.toronto.edu/~dross/ivt/RossLimLinYang_ijcv.pdf
This model is an extension of the Sequential Karhunen-Loeve Transform from:
- *A. Levy and M. Lindenbaum, Sequential Karhunen-Loeve Basis Extraction and
+ :doi:`A. Levy and M. Lindenbaum, Sequential Karhunen-Loeve Basis Extraction and
its Application to Images, IEEE Transactions on Image Processing, Volume 9,
- Number 8, pp. 1371-1374, August 2000.*
- See https://www.cs.technion.ac.il/~mic/doc/skl-ip.pdf
+ Number 8, pp. 1371-1374, August 2000. <10.1109/83.855432>`
We have specifically abstained from an optimization used by authors of both
papers, a QR decomposition used in specific situations to reduce the
diff --git a/sklearn/decomposition/_kernel_pca.py b/sklearn/decomposition/_kernel_pca.py
index 4e3ad720ae126..9f598c3eba670 100644
--- a/sklearn/decomposition/_kernel_pca.py
+++ b/sklearn/decomposition/_kernel_pca.py
@@ -216,7 +216,7 @@ class KernelPCA(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstimato
.. [2] `Bakır, Gökhan H., Jason Weston, and Bernhard Schölkopf.
"Learning to find pre-images."
Advances in neural information processing systems 16 (2004): 449-456.
- `_
+ `_
.. [3] :arxiv:`Halko, Nathan, Per-Gunnar Martinsson, and Joel A. Tropp.
"Finding structure with randomness: Probabilistic algorithms for
@@ -396,7 +396,7 @@ def _fit_inverse_transform(self, X_transformed, X):
n_samples = X_transformed.shape[0]
K = self._get_kernel(X_transformed)
K.flat[:: n_samples + 1] += self.alpha
- self.dual_coef_ = linalg.solve(K, X, sym_pos=True, overwrite_a=True)
+ self.dual_coef_ = linalg.solve(K, X, assume_a="pos", overwrite_a=True)
self.X_transformed_fit_ = X_transformed
def fit(self, X, y=None):
@@ -532,7 +532,7 @@ def inverse_transform(self, X):
`Bakır, Gökhan H., Jason Weston, and Bernhard Schölkopf.
"Learning to find pre-images."
Advances in neural information processing systems 16 (2004): 449-456.
- `_
+ `_
"""
if not self.fit_inverse_transform:
raise NotFittedError(
diff --git a/sklearn/decomposition/_lda.py b/sklearn/decomposition/_lda.py
index 6db9d900566eb..0d72f18cce1d1 100644
--- a/sklearn/decomposition/_lda.py
+++ b/sklearn/decomposition/_lda.py
@@ -292,7 +292,7 @@ class LatentDirichletAllocation(
--------
sklearn.discriminant_analysis.LinearDiscriminantAnalysis:
A classifier with a linear decision boundary, generated by fitting
- class conditional densities to the data and using Bayes’ rule.
+ class conditional densities to the data and using Bayes' rule.
References
----------
diff --git a/sklearn/decomposition/_pca.py b/sklearn/decomposition/_pca.py
index 635e119ae445d..07502c5a034cd 100644
--- a/sklearn/decomposition/_pca.py
+++ b/sklearn/decomposition/_pca.py
@@ -211,7 +211,7 @@ class PCA(_BasePCA):
.. versionadded:: 1.1
- power_iteration_normalizer : {‘auto’, ‘QR’, ‘LU’, ‘none’}, default=’auto’
+ power_iteration_normalizer : {'auto', 'QR', 'LU', 'none'}, default='auto'
Power iteration normalizer for randomized SVD solver.
Not used by ARPACK. See :func:`~sklearn.utils.extmath.randomized_svd`
for more details.
diff --git a/sklearn/decomposition/_truncated_svd.py b/sklearn/decomposition/_truncated_svd.py
index b8417543783d4..b5b44c236f46b 100644
--- a/sklearn/decomposition/_truncated_svd.py
+++ b/sklearn/decomposition/_truncated_svd.py
@@ -67,7 +67,7 @@ class TruncatedSVD(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstim
.. versionadded:: 1.1
- power_iteration_normalizer : {‘auto’, ‘QR’, ‘LU’, ‘none’}, default=’auto’
+ power_iteration_normalizer : {'auto', 'QR', 'LU', 'none'}, default='auto'
Power iteration normalizer for randomized SVD solver.
Not used by ARPACK. See :func:`~sklearn.utils.extmath.randomized_svd`
for more details.
diff --git a/sklearn/decomposition/tests/test_fastica.py b/sklearn/decomposition/tests/test_fastica.py
index 082b7d68dee79..b8c99803df373 100644
--- a/sklearn/decomposition/tests/test_fastica.py
+++ b/sklearn/decomposition/tests/test_fastica.py
@@ -71,7 +71,7 @@ def test_fastica_return_dtypes(global_dtype):
# FIXME remove filter in 1.3
@pytest.mark.filterwarnings(
- "ignore:From version 1.3 whiten='unit-variance' will be used by default."
+ "ignore:Starting in v1.3, whiten='unit-variance' will be used by default."
)
@pytest.mark.parametrize("add_noise", [True, False])
def test_fastica_simple(add_noise, global_random_seed, global_dtype):
@@ -353,7 +353,7 @@ def test_inverse_transform(
# FIXME remove filter in 1.3
@pytest.mark.filterwarnings(
- "ignore:From version 1.3 whiten='unit-variance' will be used by default."
+ "ignore:Starting in v1.3, whiten='unit-variance' will be used by default."
)
def test_fastica_errors():
n_features = 3
@@ -398,7 +398,7 @@ def test_fastica_whiten_default_value_deprecation(ica):
"""
rng = np.random.RandomState(0)
X = rng.random_sample((100, 10))
- with pytest.warns(FutureWarning, match=r"From version 1.3 whiten="):
+ with pytest.warns(FutureWarning, match=r"Starting in v1.3, whiten="):
ica.fit(X)
assert ica._whiten == "arbitrary-variance"
diff --git a/sklearn/decomposition/tests/test_nmf.py b/sklearn/decomposition/tests/test_nmf.py
index 9f3df5b64a803..930483eaa438e 100644
--- a/sklearn/decomposition/tests/test_nmf.py
+++ b/sklearn/decomposition/tests/test_nmf.py
@@ -51,13 +51,12 @@ def test_initialize_nn_output():
)
def test_parameter_checking():
A = np.ones((2, 2))
- name = "spam"
with ignore_warnings(category=FutureWarning):
# TODO remove in 1.2
msg = "Invalid regularization parameter: got 'spam' instead of one of"
with pytest.raises(ValueError, match=msg):
- NMF(regularization=name).fit(A)
+ NMF(regularization="spam").fit(A)
msg = "Invalid beta_loss parameter: solver 'cd' does not handle beta_loss = 1.0"
with pytest.raises(ValueError, match=msg):
diff --git a/sklearn/ensemble/_bagging.py b/sklearn/ensemble/_bagging.py
index adac7e063191f..6db09c3996406 100644
--- a/sklearn/ensemble/_bagging.py
+++ b/sklearn/ensemble/_bagging.py
@@ -561,7 +561,7 @@ class BaggingClassifier(ClassifierMixin, BaseBagging):
details).
- If int, then draw `max_features` features.
- - If float, then draw `max_features * X.shape[1]` features.
+ - If float, then draw `max(1, int(max_features * n_features_in_))` features.
bootstrap : bool, default=True
Whether samples are drawn with replacement. If False, sampling
@@ -993,7 +993,7 @@ class BaggingRegressor(RegressorMixin, BaseBagging):
details).
- If int, then draw `max_features` features.
- - If float, then draw `max_features * X.shape[1]` features.
+ - If float, then draw `max(1, int(max_features * n_features_in_))` features.
bootstrap : bool, default=True
Whether samples are drawn with replacement. If False, sampling
diff --git a/sklearn/ensemble/_forest.py b/sklearn/ensemble/_forest.py
index 919586001c58e..3212d6bb59457 100644
--- a/sklearn/ensemble/_forest.py
+++ b/sklearn/ensemble/_forest.py
@@ -1157,7 +1157,7 @@ class RandomForestClassifier(ForestClassifier):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `round(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=sqrt(n_features)`.
- If "sqrt", then `max_features=sqrt(n_features)`.
@@ -1518,7 +1518,7 @@ class RandomForestRegressor(ForestRegressor):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `round(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=n_features`.
- If "sqrt", then `max_features=sqrt(n_features)`.
@@ -1829,7 +1829,7 @@ class ExtraTreesClassifier(ForestClassifier):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `round(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=sqrt(n_features)`.
- If "sqrt", then `max_features=sqrt(n_features)`.
@@ -2177,7 +2177,7 @@ class ExtraTreesRegressor(ForestRegressor):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `round(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=n_features`.
- If "sqrt", then `max_features=sqrt(n_features)`.
diff --git a/sklearn/ensemble/_gb.py b/sklearn/ensemble/_gb.py
index 9b776a7feab10..316a30699b4be 100644
--- a/sklearn/ensemble/_gb.py
+++ b/sklearn/ensemble/_gb.py
@@ -991,12 +991,15 @@ def loss_(self):
class GradientBoostingClassifier(ClassifierMixin, BaseGradientBoosting):
"""Gradient Boosting for classification.
- GB builds an additive model in a
- forward stage-wise fashion; it allows for the optimization of
- arbitrary differentiable loss functions. In each stage ``n_classes_``
- regression trees are fit on the negative gradient of the loss function,
- e.g. binary or multiclass log loss. Binary classification
- is a special case where only a single regression tree is induced.
+ This algorithm builds an additive model in a forward stage-wise fashion; it
+ allows for the optimization of arbitrary differentiable loss functions. In
+ each stage ``n_classes_`` regression trees are fit on the negative gradient
+ of the loss function, e.g. binary or multiclass log loss. Binary
+ classification is a special case where only a single regression tree is
+ induced.
+
+ :class:`sklearn.ensemble.HistGradientBoostingClassifier` is a much faster
+ variant of this algorithm for intermediate datasets (`n_samples >= 10_000`).
Read more in the :ref:`User Guide `.
@@ -1122,7 +1125,7 @@ class GradientBoostingClassifier(ClassifierMixin, BaseGradientBoosting):
- If int, values must be in the range `[1, inf)`.
- If float, values must be in the range `(0.0, 1.0]` and the features
- considered at each split will be `int(max_features * n_features)`.
+ considered at each split will be `max(1, int(max_features * n_features_in_))`.
- If 'auto', then `max_features=sqrt(n_features)`.
- If 'sqrt', then `max_features=sqrt(n_features)`.
- If 'log2', then `max_features=log2(n_features)`.
@@ -1559,10 +1562,13 @@ def staged_predict_proba(self, X):
class GradientBoostingRegressor(RegressorMixin, BaseGradientBoosting):
"""Gradient Boosting for regression.
- GB builds an additive model in a forward stage-wise fashion;
- it allows for the optimization of arbitrary differentiable loss functions.
- In each stage a regression tree is fit on the negative gradient of the
- given loss function.
+ This estimator builds an additive model in a forward stage-wise fashion; it
+ allows for the optimization of arbitrary differentiable loss functions. In
+ each stage a regression tree is fit on the negative gradient of the given
+ loss function.
+
+ :class:`sklearn.ensemble.HistGradientBoostingRegressor` is a much faster
+ variant of this algorithm for intermediate datasets (`n_samples >= 10_000`).
Read more in the :ref:`User Guide `.
@@ -1695,7 +1701,7 @@ class GradientBoostingRegressor(RegressorMixin, BaseGradientBoosting):
- If int, values must be in the range `[1, inf)`.
- If float, values must be in the range `(0.0, 1.0]` and the features
- considered at each split will be `int(max_features * n_features)`.
+ considered at each split will be `max(1, int(max_features * n_features_in_))`.
- If "auto", then `max_features=n_features`.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
diff --git a/sklearn/ensemble/_iforest.py b/sklearn/ensemble/_iforest.py
index 4be74d2873b9e..a80cd31294b1c 100644
--- a/sklearn/ensemble/_iforest.py
+++ b/sklearn/ensemble/_iforest.py
@@ -78,7 +78,7 @@ class IsolationForest(OutlierMixin, BaseBagging):
The number of features to draw from X to train each base estimator.
- If int, then draw `max_features` features.
- - If float, then draw `max_features * X.shape[1]` features.
+ - If float, then draw `max(1, int(max_features * n_features_in_))` features.
bootstrap : bool, default=False
If True, individual trees are fit on random subsets of the training
diff --git a/sklearn/externals/_lobpcg.py b/sklearn/externals/_lobpcg.py
index 1de3900b3f89c..8340b322c7b3c 100644
--- a/sklearn/externals/_lobpcg.py
+++ b/sklearn/externals/_lobpcg.py
@@ -141,7 +141,7 @@ def lobpcg(
retLambdaHistory=False,
retResidualNormsHistory=False,
):
- """Locally Optimal Block Preconditioned Conjugate Gradient Method (LOBPCG)
+ """Locally Optimal Block Preconditioned Conjugate Gradient Method (LOBPCG).
LOBPCG is a preconditioned eigensolver for large symmetric positive
definite (SPD) generalized eigenproblems.
@@ -161,7 +161,7 @@ def lobpcg(
Preconditioner to `A`; by default ``M = Identity``.
`M` should approximate the inverse of `A`.
Y : ndarray, float32 or float64, optional
- n-by-sizeY matrix of constraints (non-sparse), sizeY < n
+ An n-by-sizeY matrix of constraints (non-sparse), sizeY < n.
The iterations will be performed in the B-orthogonal complement
of the column-space of Y. Y must be full rank.
tol : scalar, optional
@@ -181,7 +181,7 @@ def lobpcg(
Returns
-------
w : ndarray
- Array of ``k`` eigenvalues
+ Array of ``k`` eigenvalues.
v : ndarray
An array of ``k`` eigenvectors. `v` has the same shape as `X`.
lambdas : list of ndarray, optional
@@ -240,7 +240,6 @@ def lobpcg(
Examples
--------
-
Solve ``A x = lambda x`` with constraints and preconditioning.
>>> import numpy as np
@@ -293,7 +292,6 @@ def lobpcg(
Note that the vectors passed in Y are the eigenvectors of the 3 smallest
eigenvalues. The results returned are orthogonal to those.
-
"""
blockVectorX = X
blockVectorY = Y
diff --git a/sklearn/feature_extraction/image.py b/sklearn/feature_extraction/image.py
index 031a597591afc..515a7990306b6 100644
--- a/sklearn/feature_extraction/image.py
+++ b/sklearn/feature_extraction/image.py
@@ -321,7 +321,7 @@ def _extract_patches(arr, patch_shape=8, extraction_step=1):
def extract_patches_2d(image, patch_size, *, max_patches=None, random_state=None):
- """Reshape a 2D image into a collection of patches
+ """Reshape a 2D image into a collection of patches.
The resulting patches are allocated in a dedicated array.
diff --git a/sklearn/feature_extraction/text.py b/sklearn/feature_extraction/text.py
index b565aeadc53c8..6f033d9b8b6d6 100644
--- a/sklearn/feature_extraction/text.py
+++ b/sklearn/feature_extraction/text.py
@@ -1519,6 +1519,7 @@ class TfidfTransformer(_OneToOneFeatureMixin, TransformerMixin, BaseEstimator):
been applied.
- 'l1': Sum of absolute values of vector elements is 1.
See :func:`preprocessing.normalize`.
+ - None: No normalization.
use_idf : bool, default=True
Enable inverse-document-frequency reweighting. If False, idf(t) = 1.
diff --git a/sklearn/feature_selection/_from_model.py b/sklearn/feature_selection/_from_model.py
index 0c41c66fbef1f..7c78902fcb172 100644
--- a/sklearn/feature_selection/_from_model.py
+++ b/sklearn/feature_selection/_from_model.py
@@ -22,9 +22,13 @@ def _calculate_threshold(estimator, importances, threshold):
if threshold is None:
# determine default from estimator
est_name = estimator.__class__.__name__
- if (
- hasattr(estimator, "penalty") and estimator.penalty == "l1"
- ) or "Lasso" in est_name:
+ is_l1_penalized = hasattr(estimator, "penalty") and estimator.penalty == "l1"
+ is_lasso = "Lasso" in est_name
+ is_elasticnet_l1_penalized = "ElasticNet" in est_name and (
+ (hasattr(estimator, "l1_ratio_") and np.isclose(estimator.l1_ratio_, 1.0))
+ or (hasattr(estimator, "l1_ratio") and np.isclose(estimator.l1_ratio, 1.0))
+ )
+ if is_l1_penalized or is_lasso or is_elasticnet_l1_penalized:
# the natural default threshold is 0 when l1 penalty was used
threshold = 1e-5
else:
@@ -93,9 +97,9 @@ class SelectFromModel(MetaEstimatorMixin, SelectorMixin, BaseEstimator):
threshold : str or float, default=None
The threshold value to use for feature selection. Features whose
- importance is greater or equal are kept while the others are
- discarded. If "median" (resp. "mean"), then the ``threshold`` value is
- the median (resp. the mean) of the feature importances. A scaling
+ absolute importance value is greater or equal are kept while the others
+ are discarded. If "median" (resp. "mean"), then the ``threshold`` value
+ is the median (resp. the mean) of the feature importances. A scaling
factor (e.g., "1.25*mean") may also be used. If None and if the
estimator has a parameter penalty set to l1, either explicitly
or implicitly (e.g, Lasso), the threshold used is 1e-5.
diff --git a/sklearn/feature_selection/_rfe.py b/sklearn/feature_selection/_rfe.py
index 0f82e1775ee15..877b3bed1558a 100644
--- a/sklearn/feature_selection/_rfe.py
+++ b/sklearn/feature_selection/_rfe.py
@@ -333,7 +333,7 @@ def _fit(self, X, y, step_score=None, **fit_params):
@available_if(_estimator_has("predict"))
def predict(self, X):
- """Reduce X to the selected features and then predict using the underlying estimator.
+ """Reduce X to the selected features and predict using the estimator.
Parameters
----------
@@ -350,7 +350,7 @@ def predict(self, X):
@available_if(_estimator_has("score"))
def score(self, X, y, **fit_params):
- """Reduce X to the selected features and return the score of the underlying estimator.
+ """Reduce X to the selected features and return the score of the estimator.
Parameters
----------
@@ -448,7 +448,7 @@ def _more_tags(self):
class RFECV(RFE):
- """Recursive feature elimination with cross-validation to select the number of features.
+ """Recursive feature elimination with cross-validation to select features.
See glossary entry for :term:`cross-validation estimator`.
@@ -550,13 +550,13 @@ class RFECV(RFE):
cv_results_ : dict of ndarrays
A dict with keys:
- split(k)_test_score : ndarray of shape (n_features,)
+ split(k)_test_score : ndarray of shape (n_subsets_of_features,)
The cross-validation scores across (k)th fold.
- mean_test_score : ndarray of shape (n_features,)
+ mean_test_score : ndarray of shape (n_subsets_of_features,)
Mean of scores over the folds.
- std_test_score : ndarray of shape (n_features,)
+ std_test_score : ndarray of shape (n_subsets_of_features,)
Standard deviation of scores over the folds.
.. versionadded:: 1.0
diff --git a/sklearn/feature_selection/_univariate_selection.py b/sklearn/feature_selection/_univariate_selection.py
index 7754ea3bea7f4..920eb2220e84a 100644
--- a/sklearn/feature_selection/_univariate_selection.py
+++ b/sklearn/feature_selection/_univariate_selection.py
@@ -39,7 +39,7 @@ def _clean_nans(scores):
# Contrary to the scipy.stats.f_oneway implementation it does not
# copy the data while keeping the inputs unchanged.
def f_oneway(*args):
- """Performs a 1-way ANOVA.
+ """Perform a 1-way ANOVA.
The one-way ANOVA tests the null hypothesis that 2 or more groups have
the same population mean. The test is applied to samples from two or
@@ -50,7 +50,7 @@ def f_oneway(*args):
Parameters
----------
*args : {array-like, sparse matrix}
- sample1, sample2... The sample measurements should be given as
+ Sample1, sample2... The sample measurements should be given as
arguments.
Returns
@@ -81,13 +81,11 @@ def f_oneway(*args):
References
----------
-
.. [1] Lowry, Richard. "Concepts and Applications of Inferential
Statistics". Chapter 14.
http://faculty.vassar.edu/lowry/ch14pt1.html
.. [2] Heiman, G.W. Research Methods in Statistics. 2002.
-
"""
n_classes = len(args)
args = [as_float_array(a) for a in args]
@@ -198,14 +196,14 @@ def chi2(X, y):
p_values : ndarray of shape (n_features,)
P-values for each feature.
- Notes
- -----
- Complexity of this algorithm is O(n_classes * n_features).
-
See Also
--------
f_classif : ANOVA F-value between label/feature for classification tasks.
f_regression : F-value between label/feature for regression tasks.
+
+ Notes
+ -----
+ Complexity of this algorithm is O(n_classes * n_features).
"""
# XXX: we might want to do some of the following in logspace instead for
diff --git a/sklearn/feature_selection/tests/test_from_model.py b/sklearn/feature_selection/tests/test_from_model.py
index de45d9e0ab6a4..d193df981e4a6 100644
--- a/sklearn/feature_selection/tests/test_from_model.py
+++ b/sklearn/feature_selection/tests/test_from_model.py
@@ -14,7 +14,14 @@
from sklearn.cross_decomposition import CCA, PLSCanonical, PLSRegression
from sklearn.datasets import make_friedman1
from sklearn.exceptions import NotFittedError
-from sklearn.linear_model import LogisticRegression, SGDClassifier, Lasso
+from sklearn.linear_model import (
+ LogisticRegression,
+ SGDClassifier,
+ Lasso,
+ LassoCV,
+ ElasticNet,
+ ElasticNetCV,
+)
from sklearn.svm import LinearSVC
from sklearn.feature_selection import SelectFromModel
from sklearn.ensemble import RandomForestClassifier, HistGradientBoostingClassifier
@@ -318,7 +325,16 @@ def test_sample_weight():
assert np.all(weighted_mask == reweighted_mask)
-def test_coef_default_threshold():
+@pytest.mark.parametrize(
+ "estimator",
+ [
+ Lasso(alpha=0.1, random_state=42),
+ LassoCV(random_state=42),
+ ElasticNet(l1_ratio=1, random_state=42),
+ ElasticNetCV(l1_ratio=[1], random_state=42),
+ ],
+)
+def test_coef_default_threshold(estimator):
X, y = datasets.make_classification(
n_samples=100,
n_features=10,
@@ -330,7 +346,7 @@ def test_coef_default_threshold():
)
# For the Lasso and related models, the threshold defaults to 1e-5
- transformer = SelectFromModel(estimator=Lasso(alpha=0.1, random_state=42))
+ transformer = SelectFromModel(estimator=estimator)
transformer.fit(X, y)
X_new = transformer.transform(X)
mask = np.abs(transformer.estimator_.coef_) > 1e-5
diff --git a/sklearn/gaussian_process/kernels.py b/sklearn/gaussian_process/kernels.py
index 4e36dfa7add42..3ac3866cf9ba7 100644
--- a/sklearn/gaussian_process/kernels.py
+++ b/sklearn/gaussian_process/kernels.py
@@ -1421,7 +1421,7 @@ def __repr__(self):
class RBF(StationaryKernelMixin, NormalizedKernelMixin, Kernel):
- """Radial-basis function kernel (aka squared-exponential kernel).
+ """Radial basis function kernel (aka squared-exponential kernel).
The RBF kernel is a stationary kernel. It is also known as the
"squared exponential" kernel. It is parameterized by a length scale
diff --git a/sklearn/impute/_base.py b/sklearn/impute/_base.py
index 0c8a6f2c07a21..73e20c2cd5101 100644
--- a/sklearn/impute/_base.py
+++ b/sklearn/impute/_base.py
@@ -9,9 +9,9 @@
import numpy as np
import numpy.ma as ma
from scipy import sparse as sp
-from scipy import stats
from ..base import BaseEstimator, TransformerMixin
+from ..utils.fixes import _mode
from ..utils.sparsefuncs import _get_median
from ..utils.validation import check_is_fitted
from ..utils.validation import FLOAT_DTYPES
@@ -51,7 +51,7 @@ def _most_frequent(array, extra_value, n_repeat):
if count == most_frequent_count
)
else:
- mode = stats.mode(array)
+ mode = _mode(array)
most_frequent_value = mode[0][0]
most_frequent_count = mode[1][0]
else:
@@ -130,7 +130,10 @@ def _more_tags(self):
class SimpleImputer(_BaseImputer):
- """Imputation transformer for completing missing values.
+ """Univariate imputer for completing missing values with simple strategies.
+
+ Replace missing values using a descriptive statistic (e.g. mean, median, or
+ most frequent) along each column, or using a constant value.
Read more in the :ref:`User Guide `.
@@ -218,13 +221,21 @@ class SimpleImputer(_BaseImputer):
See Also
--------
- IterativeImputer : Multivariate imputation of missing values.
+ IterativeImputer : Multivariate imputer that estimates values to impute for
+ each feature with missing values from all the others.
+ KNNImputer : Multivariate imputer that estimates missing features using
+ nearest samples.
Notes
-----
Columns which only contained missing values at :meth:`fit` are discarded
upon :meth:`transform` if strategy is not `"constant"`.
+ In a prediction context, simple imputation usually performs poorly when
+ associated with a weak learner. However, with a powerful learner, it can
+ lead to as good or better performance than complex imputation such as
+ :class:`~sklearn.impute.IterativeImputer` or :class:`~sklearn.impute.KNNImputer`.
+
Examples
--------
>>> import numpy as np
@@ -278,6 +289,10 @@ def _validate_input(self, X, in_fit):
else:
dtype = FLOAT_DTYPES
+ if not in_fit and self._fit_dtype.kind == "O":
+ # Use object dtype if fitted on object dtypes
+ dtype = self._fit_dtype
+
if _is_pandas_na(self.missing_values) or is_scalar_nan(self.missing_values):
force_all_finite = "allow-nan"
else:
@@ -303,6 +318,10 @@ def _validate_input(self, X, in_fit):
else:
raise ve
+ if in_fit:
+ # Use the dtype seen in `fit` for non-`fit` conversion
+ self._fit_dtype = X.dtype
+
_check_inputs_dtype(X, self.missing_values)
if X.dtype.kind not in ("i", "u", "f", "O"):
raise ValueError(
diff --git a/sklearn/impute/_iterative.py b/sklearn/impute/_iterative.py
index f6c32a6818455..12032c89d949d 100644
--- a/sklearn/impute/_iterative.py
+++ b/sklearn/impute/_iterative.py
@@ -183,7 +183,10 @@ class IterativeImputer(_BaseImputer):
See Also
--------
- SimpleImputer : Univariate imputation of missing values.
+ SimpleImputer : Univariate imputer for completing missing values
+ with simple strategies.
+ KNNImputer : Multivariate imputer that estimates missing features using
+ nearest samples.
Notes
-----
@@ -194,6 +197,16 @@ class IterativeImputer(_BaseImputer):
Features which contain all missing values at :meth:`fit` are discarded upon
:meth:`transform`.
+ Using defaults, the imputer scales in :math:`\\mathcal{O}(knp^3\\min(n,p))`
+ where :math:`k` = `max_iter`, :math:`n` the number of samples and
+ :math:`p` the number of features. It thus becomes prohibitively costly when
+ the number of features increases. Setting
+ `n_nearest_features << n_features`, `skip_complete=True` or increasing `tol`
+ can help to reduce its computational cost.
+
+ Depending on the nature of missing values, simple imputers can be
+ preferable in a prediction context.
+
References
----------
.. [1] `Stef van Buuren, Karin Groothuis-Oudshoorn (2011). "mice:
diff --git a/sklearn/impute/_knn.py b/sklearn/impute/_knn.py
index 497bcfafb074a..cac960e9a3436 100644
--- a/sklearn/impute/_knn.py
+++ b/sklearn/impute/_knn.py
@@ -90,10 +90,10 @@ class KNNImputer(_BaseImputer):
See Also
--------
- SimpleImputer : Imputation transformer for completing missing values
+ SimpleImputer : Univariate imputer for completing missing values
with simple strategies.
- IterativeImputer : Multivariate imputer that estimates each feature
- from all the others.
+ IterativeImputer : Multivariate imputer that estimates values to impute for
+ each feature with missing values from all the others.
References
----------
diff --git a/sklearn/impute/tests/test_impute.py b/sklearn/impute/tests/test_impute.py
index dc585571124b5..512558d28851d 100644
--- a/sklearn/impute/tests/test_impute.py
+++ b/sklearn/impute/tests/test_impute.py
@@ -1618,3 +1618,29 @@ def test_missing_indicator_feature_names_out():
feature_names = indicator.get_feature_names_out()
expected_names = ["missingindicator_a", "missingindicator_b", "missingindicator_d"]
assert_array_equal(expected_names, feature_names)
+
+
+def test_imputer_lists_fit_transform():
+ """Check transform uses object dtype when fitted on an object dtype.
+
+ Non-regression test for #19572.
+ """
+
+ X = [["a", "b"], ["c", "b"], ["a", "a"]]
+ imp_frequent = SimpleImputer(strategy="most_frequent").fit(X)
+ X_trans = imp_frequent.transform([[np.nan, np.nan]])
+ assert X_trans.dtype == object
+ assert_array_equal(X_trans, [["a", "b"]])
+
+
+@pytest.mark.parametrize("dtype_test", [np.float32, np.float64])
+def test_imputer_transform_preserves_numeric_dtype(dtype_test):
+ """Check transform preserves numeric dtype independent of fit dtype."""
+ X = np.asarray(
+ [[1.2, 3.4, np.nan], [np.nan, 1.2, 1.3], [4.2, 2, 1]], dtype=np.float64
+ )
+ imp = SimpleImputer().fit(X)
+
+ X_test = np.asarray([[np.nan, np.nan, np.nan]], dtype=dtype_test)
+ X_trans = imp.transform(X_test)
+ assert X_trans.dtype == dtype_test
diff --git a/sklearn/linear_model/_base.py b/sklearn/linear_model/_base.py
index 5e8417de21996..de747ef0850df 100644
--- a/sklearn/linear_model/_base.py
+++ b/sklearn/linear_model/_base.py
@@ -743,7 +743,7 @@ def rmatvec(b):
def _check_precomputed_gram_matrix(
- X, precompute, X_offset, X_scale, rtol=1e-7, atol=1e-5
+ X, precompute, X_offset, X_scale, rtol=None, atol=1e-5
):
"""Computes a single element of the gram matrix and compares it to
the corresponding element of the user supplied gram matrix.
@@ -764,8 +764,10 @@ def _check_precomputed_gram_matrix(
X_scale : ndarray of shape (n_features,)
Array of feature scale factors used to normalize design matrix.
- rtol : float, default=1e-7
- Relative tolerance; see numpy.allclose.
+ rtol : float, default=None
+ Relative tolerance; see numpy.allclose
+ If None, it is set to 1e-4 for arrays of dtype numpy.float32 and 1e-7
+ otherwise.
atol : float, default=1e-5
absolute tolerance; see :func`numpy.allclose`. Note that the default
@@ -788,6 +790,11 @@ def _check_precomputed_gram_matrix(
expected = np.dot(v1, v2)
actual = precompute[f1, f2]
+ dtypes = [precompute.dtype, expected.dtype]
+ if rtol is None:
+ rtols = [1e-4 if dtype == np.float32 else 1e-7 for dtype in dtypes]
+ rtol = max(rtols)
+
if not np.isclose(expected, actual, rtol=rtol, atol=atol):
raise ValueError(
"Gram matrix passed in via 'precompute' parameter "
diff --git a/sklearn/linear_model/_huber.py b/sklearn/linear_model/_huber.py
index 3fdf5aa73743f..afadc8c1efc2e 100644
--- a/sklearn/linear_model/_huber.py
+++ b/sklearn/linear_model/_huber.py
@@ -124,18 +124,19 @@ def _huber_loss_and_gradient(w, X, y, epsilon, alpha, sample_weight=None):
class HuberRegressor(LinearModel, RegressorMixin, BaseEstimator):
- """Linear regression model that is robust to outliers.
+ """L2-regularized linear regression model that is robust to outliers.
The Huber Regressor optimizes the squared loss for the samples where
- ``|(y - X'w) / sigma| < epsilon`` and the absolute loss for the samples
- where ``|(y - X'w) / sigma| > epsilon``, where w and sigma are parameters
+ ``|(y - Xw - c) / sigma| < epsilon`` and the absolute loss for the samples
+ where ``|(y - Xw - c) / sigma| > epsilon``, where the model coefficients
+ ``w``, the intercept ``c`` and the scale ``sigma`` are parameters
to be optimized. The parameter sigma makes sure that if y is scaled up
or down by a certain factor, one does not need to rescale epsilon to
achieve the same robustness. Note that this does not take into account
the fact that the different features of X may be of different scales.
- This makes sure that the loss function is not heavily influenced by the
- outliers while not completely ignoring their effect.
+ The Huber loss function has the advantage of not being heavily influenced
+ by the outliers while not completely ignoring their effect.
Read more in the :ref:`User Guide `
@@ -153,7 +154,9 @@ class HuberRegressor(LinearModel, RegressorMixin, BaseEstimator):
``scipy.optimize.minimize(method="L-BFGS-B")`` should run for.
alpha : float, default=0.0001
- Regularization parameter.
+ Strength of the squared L2 regularization. Note that the penalty is
+ equal to ``alpha * ||w||^2``.
+ Must be in the range `[0, inf)`.
warm_start : bool, default=False
This is useful if the stored attributes of a previously used model
@@ -173,13 +176,13 @@ class HuberRegressor(LinearModel, RegressorMixin, BaseEstimator):
Attributes
----------
coef_ : array, shape (n_features,)
- Features got by optimizing the Huber loss.
+ Features got by optimizing the L2-regularized Huber loss.
intercept_ : float
Bias.
scale_ : float
- The value by which ``|y - X'w - c|`` is scaled down.
+ The value by which ``|y - Xw - c|`` is scaled down.
n_features_in_ : int
Number of features seen during :term:`fit`.
diff --git a/sklearn/linear_model/_linear_loss.py b/sklearn/linear_model/_linear_loss.py
index 64a99325dcd7a..7623a7fb20838 100644
--- a/sklearn/linear_model/_linear_loss.py
+++ b/sklearn/linear_model/_linear_loss.py
@@ -327,6 +327,8 @@ def gradient_hessian_product(
# Calculate the double derivative with respect to intercept.
# Note: In case hX is sparse, hX.sum is a matrix object.
hX_sum = np.squeeze(np.asarray(hX.sum(axis=0)))
+ # prevent squeezing to zero-dim array if n_features == 1
+ hX_sum = np.atleast_1d(hX_sum)
# With intercept included and l2_reg_strength = 0, hessp returns
# res = (X, 1)' @ diag(h) @ (X, 1) @ s
diff --git a/sklearn/linear_model/_logistic.py b/sklearn/linear_model/_logistic.py
index 72b602e409801..93f6e31b12223 100644
--- a/sklearn/linear_model/_logistic.py
+++ b/sklearn/linear_model/_logistic.py
@@ -872,8 +872,8 @@ class LogisticRegression(LinearClassifierMixin, SparseCoefMixin, BaseEstimator):
.. seealso::
Refer to the User Guide for more information regarding
:class:`LogisticRegression` and more specifically the
- `Table `_
- summarazing solver/penalty supports.
+ :ref:`Table `
+ summarizing solver/penalty supports.
.. versionadded:: 0.17
Stochastic Average Gradient descent solver.
diff --git a/sklearn/linear_model/_omp.py b/sklearn/linear_model/_omp.py
index b86c35c41de85..c1e3762343628 100644
--- a/sklearn/linear_model/_omp.py
+++ b/sklearn/linear_model/_omp.py
@@ -361,7 +361,7 @@ def orthogonal_mp(
Orthogonal matching pursuit was introduced in S. Mallat, Z. Zhang,
Matching pursuits with time-frequency dictionaries, IEEE Transactions on
Signal Processing, Vol. 41, No. 12. (December 1993), pp. 3397-3415.
- (http://blanche.polytechnique.fr/~mallat/papiers/MallatPursuit93.pdf)
+ (https://www.di.ens.fr/~mallat/papiers/MallatPursuit93.pdf)
This implementation is based on Rubinstein, R., Zibulevsky, M. and Elad,
M., Efficient Implementation of the K-SVD Algorithm using Batch Orthogonal
@@ -515,7 +515,7 @@ def orthogonal_mp_gram(
Orthogonal matching pursuit was introduced in G. Mallat, Z. Zhang,
Matching pursuits with time-frequency dictionaries, IEEE Transactions on
Signal Processing, Vol. 41, No. 12. (December 1993), pp. 3397-3415.
- (http://blanche.polytechnique.fr/~mallat/papiers/MallatPursuit93.pdf)
+ (https://www.di.ens.fr/~mallat/papiers/MallatPursuit93.pdf)
This implementation is based on Rubinstein, R., Zibulevsky, M. and Elad,
M., Efficient Implementation of the K-SVD Algorithm using Batch Orthogonal
@@ -670,7 +670,7 @@ class OrthogonalMatchingPursuit(MultiOutputMixin, RegressorMixin, LinearModel):
Orthogonal matching pursuit was introduced in G. Mallat, Z. Zhang,
Matching pursuits with time-frequency dictionaries, IEEE Transactions on
Signal Processing, Vol. 41, No. 12. (December 1993), pp. 3397-3415.
- (http://blanche.polytechnique.fr/~mallat/papiers/MallatPursuit93.pdf)
+ (https://www.di.ens.fr/~mallat/papiers/MallatPursuit93.pdf)
This implementation is based on Rubinstein, R., Zibulevsky, M. and Elad,
M., Efficient Implementation of the K-SVD Algorithm using Batch Orthogonal
diff --git a/sklearn/linear_model/_ransac.py b/sklearn/linear_model/_ransac.py
index 8d20005430769..5ce0c7b1735df 100644
--- a/sklearn/linear_model/_ransac.py
+++ b/sklearn/linear_model/_ransac.py
@@ -228,7 +228,7 @@ class RANSACRegressor(
References
----------
.. [1] https://en.wikipedia.org/wiki/RANSAC
- .. [2] https://www.sri.com/sites/default/files/publications/ransac-publication.pdf
+ .. [2] https://www.sri.com/wp-content/uploads/2021/12/ransac-publication.pdf
.. [3] http://www.bmva.org/bmvc/2009/Papers/Paper355/Paper355.pdf
Examples
diff --git a/sklearn/linear_model/_ridge.py b/sklearn/linear_model/_ridge.py
index dee703b73c059..a3fc5cdc82baf 100644
--- a/sklearn/linear_model/_ridge.py
+++ b/sklearn/linear_model/_ridge.py
@@ -209,12 +209,12 @@ def _solve_cholesky(X, y, alpha):
if one_alpha:
A.flat[:: n_features + 1] += alpha[0]
- return linalg.solve(A, Xy, sym_pos=True, overwrite_a=True).T
+ return linalg.solve(A, Xy, assume_a="pos", overwrite_a=True).T
else:
coefs = np.empty([n_targets, n_features], dtype=X.dtype)
for coef, target, current_alpha in zip(coefs, Xy.T, alpha):
A.flat[:: n_features + 1] += current_alpha
- coef[:] = linalg.solve(A, target, sym_pos=True, overwrite_a=False).ravel()
+ coef[:] = linalg.solve(A, target, assume_a="pos", overwrite_a=False).ravel()
A.flat[:: n_features + 1] -= current_alpha
return coefs
@@ -246,7 +246,7 @@ def _solve_cholesky_kernel(K, y, alpha, sample_weight=None, copy=False):
# Note: we must use overwrite_a=False in order to be able to
# use the fall-back solution below in case a LinAlgError
# is raised
- dual_coef = linalg.solve(K, y, sym_pos=True, overwrite_a=False)
+ dual_coef = linalg.solve(K, y, assume_a="pos", overwrite_a=False)
except np.linalg.LinAlgError:
warnings.warn(
"Singular matrix in solving dual problem. Using "
@@ -270,7 +270,7 @@ def _solve_cholesky_kernel(K, y, alpha, sample_weight=None, copy=False):
K.flat[:: n_samples + 1] += current_alpha
dual_coef[:] = linalg.solve(
- K, target, sym_pos=True, overwrite_a=False
+ K, target, assume_a="pos", overwrite_a=False
).ravel()
K.flat[:: n_samples + 1] -= current_alpha
diff --git a/sklearn/linear_model/tests/test_coordinate_descent.py b/sklearn/linear_model/tests/test_coordinate_descent.py
index e5d7ba358c1f5..124da4f921e00 100644
--- a/sklearn/linear_model/tests/test_coordinate_descent.py
+++ b/sklearn/linear_model/tests/test_coordinate_descent.py
@@ -1065,6 +1065,29 @@ def test_elasticnet_precompute_gram_weighted_samples():
assert_allclose(clf1.coef_, clf2.coef_)
+def test_elasticnet_precompute_gram():
+ # Check the dtype-aware check for a precomputed Gram matrix
+ # (see https://github.com/scikit-learn/scikit-learn/pull/22059
+ # and https://github.com/scikit-learn/scikit-learn/issues/21997).
+ # Here: (X_c.T, X_c)[2, 3] is not equal to np.dot(X_c[:, 2], X_c[:, 3])
+ # but within tolerance for np.float32
+
+ rng = np.random.RandomState(58)
+ X = rng.binomial(1, 0.25, (1000, 4)).astype(np.float32)
+ y = rng.rand(1000).astype(np.float32)
+
+ X_c = X - np.average(X, axis=0)
+ gram = np.dot(X_c.T, X_c)
+
+ clf1 = ElasticNet(alpha=0.01, precompute=gram)
+ clf1.fit(X_c, y)
+
+ clf2 = ElasticNet(alpha=0.01, precompute=False)
+ clf2.fit(X, y)
+
+ assert_allclose(clf1.coef_, clf2.coef_)
+
+
def test_warm_start_convergence():
X, y, _, _ = build_dataset()
model = ElasticNet(alpha=1e-3, tol=1e-3).fit(X, y)
diff --git a/sklearn/linear_model/tests/test_logistic.py b/sklearn/linear_model/tests/test_logistic.py
index 5bb2b83094290..4c8c9daf78731 100644
--- a/sklearn/linear_model/tests/test_logistic.py
+++ b/sklearn/linear_model/tests/test_logistic.py
@@ -2071,3 +2071,13 @@ def test_large_sparse_matrix(solver):
LogisticRegression(solver=solver).fit(X, y)
else:
LogisticRegression(solver=solver).fit(X, y)
+
+
+def test_single_feature_newton_cg():
+ # Test that Newton-CG works with a single feature and intercept.
+ # Non-regression test for issue #23605.
+
+ X = np.array([[0.5, 0.65, 1.1, 1.25, 0.8, 0.54, 0.95, 0.7]]).T
+ y = np.array([1, 1, 0, 0, 1, 1, 0, 1])
+ assert X.shape[1] == 1
+ LogisticRegression(solver="newton-cg", fit_intercept=True).fit(X, y)
diff --git a/sklearn/manifold/_locally_linear.py b/sklearn/manifold/_locally_linear.py
index a9c6ec350b912..f5f64dad2c1a8 100644
--- a/sklearn/manifold/_locally_linear.py
+++ b/sklearn/manifold/_locally_linear.py
@@ -72,7 +72,7 @@ def barycenter_weights(X, Y, indices, reg=1e-3):
else:
R = reg
G.flat[:: n_neighbors + 1] += R
- w = solve(G, v, sym_pos=True)
+ w = solve(G, v, assume_a="pos")
B[i, :] = w / np.sum(w)
return B
diff --git a/sklearn/manifold/_mds.py b/sklearn/manifold/_mds.py
index 930f8d19b7b5e..a848b5d63c5a8 100644
--- a/sklearn/manifold/_mds.py
+++ b/sklearn/manifold/_mds.py
@@ -36,6 +36,8 @@ def _smacof_single(
metric : bool, default=True
Compute metric or nonmetric SMACOF algorithm.
+ When ``False`` (i.e. non-metric MDS), dissimilarities with 0 are considered as
+ missing values.
n_components : int, default=2
Number of dimensions in which to immerse the dissimilarities. If an
@@ -180,6 +182,8 @@ def smacof(
metric : bool, default=True
Compute metric or nonmetric SMACOF algorithm.
+ When ``False`` (i.e. non-metric MDS), dissimilarities with 0 are considered as
+ missing values.
n_components : int, default=2
Number of dimensions in which to immerse the dissimilarities. If an
@@ -318,6 +322,8 @@ class MDS(BaseEstimator):
metric : bool, default=True
If ``True``, perform metric MDS; otherwise, perform nonmetric MDS.
+ When ``False`` (i.e. non-metric MDS), dissimilarities with 0 are considered as
+ missing values.
n_init : int, default=4
Number of times the SMACOF algorithm will be run with different
@@ -491,7 +497,7 @@ def fit_transform(self, X, y=None, init=None):
y : Ignored
Not used, present for API consistency by convention.
- init : ndarray of shape (n_samples,), default=None
+ init : ndarray of shape (n_samples, n_components), default=None
Starting configuration of the embedding to initialize the SMACOF
algorithm. By default, the algorithm is initialized with a randomly
chosen array.
diff --git a/sklearn/manifold/_t_sne.py b/sklearn/manifold/_t_sne.py
index 5b7a3c4efd753..8e7b7f12cc59a 100644
--- a/sklearn/manifold/_t_sne.py
+++ b/sklearn/manifold/_t_sne.py
@@ -564,7 +564,8 @@ class TSNE(BaseEstimator):
is used in other manifold learning algorithms. Larger datasets
usually require a larger perplexity. Consider selecting a value
between 5 and 50. Different values can result in significantly
- different results.
+ different results. The perplexity must be less that the number
+ of samples.
early_exaggeration : float, default=12.0
Controls how tight natural clusters in the original space are in
@@ -739,7 +740,7 @@ class TSNE(BaseEstimator):
>>> from sklearn.manifold import TSNE
>>> X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
>>> X_embedded = TSNE(n_components=2, learning_rate='auto',
- ... init='random').fit_transform(X)
+ ... init='random', perplexity=3).fit_transform(X)
>>> X_embedded.shape
(4, 2)
"""
@@ -787,6 +788,10 @@ def __init__(
self.n_jobs = n_jobs
self.square_distances = square_distances
+ def _check_params_vs_input(self, X):
+ if self.perplexity >= X.shape[0]:
+ raise ValueError("perplexity must be less than n_samples")
+
def _fit(self, X, skip_num_points=0):
"""Private function to fit the model using X as training data."""
@@ -1114,6 +1119,7 @@ def fit_transform(self, X, y=None):
X_new : ndarray of shape (n_samples, n_components)
Embedding of the training data in low-dimensional space.
"""
+ self._check_params_vs_input(X)
embedding = self._fit(X)
self.embedding_ = embedding
return self.embedding_
diff --git a/sklearn/manifold/tests/test_t_sne.py b/sklearn/manifold/tests/test_t_sne.py
index 861500e4a8891..997da9e542fda 100644
--- a/sklearn/manifold/tests/test_t_sne.py
+++ b/sklearn/manifold/tests/test_t_sne.py
@@ -389,7 +389,7 @@ def test_trustworthiness_not_euclidean_metric():
@pytest.mark.filterwarnings("ignore:The default initialization in TSNE")
def test_early_exaggeration_too_small():
# Early exaggeration factor must be >= 1.
- tsne = TSNE(early_exaggeration=0.99)
+ tsne = TSNE(early_exaggeration=0.99, perplexity=1)
with pytest.raises(ValueError, match="early_exaggeration .*"):
tsne.fit_transform(np.array([[0.0], [0.0]]))
@@ -398,7 +398,7 @@ def test_early_exaggeration_too_small():
@pytest.mark.filterwarnings("ignore:The default initialization in TSNE")
def test_too_few_iterations():
# Number of gradient descent iterations must be at least 200.
- tsne = TSNE(n_iter=199)
+ tsne = TSNE(n_iter=199, perplexity=1)
with pytest.raises(ValueError, match="n_iter .*"):
tsne.fit_transform(np.array([[0.0], [0.0]]))
@@ -425,6 +425,7 @@ def test_bad_precomputed_distances(method, D, retype, message_regex):
method=method,
init="random",
random_state=42,
+ perplexity=1,
)
with pytest.raises(ValueError, match=message_regex):
tsne.fit_transform(retype(D))
@@ -437,6 +438,7 @@ def test_exact_no_precomputed_sparse():
method="exact",
init="random",
random_state=42,
+ perplexity=1,
)
with pytest.raises(TypeError, match="sparse"):
tsne.fit_transform(sp.csr_matrix([[0, 5], [5, 0]]))
@@ -447,7 +449,7 @@ def test_high_perplexity_precomputed_sparse_distances():
# Perplexity should be less than 50
dist = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [1.0, 0.0, 0.0]])
bad_dist = sp.csr_matrix(dist)
- tsne = TSNE(metric="precomputed", init="random", random_state=42)
+ tsne = TSNE(metric="precomputed", init="random", random_state=42, perplexity=1)
msg = "3 neighbors per samples are required, but some samples have only 1"
with pytest.raises(ValueError, match=msg):
tsne.fit_transform(bad_dist)
@@ -482,7 +484,7 @@ def metric(x, y):
return -1
# Negative computed distances should be caught even if result is squared
- tsne = TSNE(metric=metric, method="exact")
+ tsne = TSNE(metric=metric, method="exact", perplexity=1)
X = np.array([[0.0, 0.0], [1.0, 1.0]])
with pytest.raises(ValueError, match="All distances .*metric given.*"):
tsne.fit_transform(X)
@@ -491,7 +493,7 @@ def metric(x, y):
@pytest.mark.filterwarnings("ignore:The default learning rate in TSNE")
def test_init_not_available():
# 'init' must be 'pca', 'random', or numpy array.
- tsne = TSNE(init="not available")
+ tsne = TSNE(init="not available", perplexity=1)
m = "'init' must be 'pca', 'random', or a numpy array"
with pytest.raises(ValueError, match=m):
tsne.fit_transform(np.array([[0.0], [1.0]]))
@@ -519,11 +521,11 @@ def test_init_ndarray_precomputed():
@pytest.mark.filterwarnings("ignore:The default initialization in TSNE")
def test_distance_not_available():
# 'metric' must be valid.
- tsne = TSNE(metric="not available", method="exact")
+ tsne = TSNE(metric="not available", method="exact", perplexity=1)
with pytest.raises(ValueError, match="Unknown metric not available.*"):
tsne.fit_transform(np.array([[0.0], [1.0]]))
- tsne = TSNE(metric="not available", method="barnes_hut")
+ tsne = TSNE(metric="not available", method="barnes_hut", perplexity=1)
with pytest.raises(ValueError, match="Metric 'not available' not valid.*"):
tsne.fit_transform(np.array([[0.0], [1.0]]))
@@ -532,7 +534,7 @@ def test_distance_not_available():
@pytest.mark.filterwarnings("ignore:The default initialization in TSNE")
def test_method_not_available():
# 'nethod' must be 'barnes_hut' or 'exact'
- tsne = TSNE(method="not available")
+ tsne = TSNE(method="not available", perplexity=1)
with pytest.raises(ValueError, match="'method' must be 'barnes_hut' or "):
tsne.fit_transform(np.array([[0.0], [1.0]]))
@@ -542,7 +544,7 @@ def test_method_not_available():
def test_angle_out_of_range_checks():
# check the angle parameter range
for angle in [-1, -1e-6, 1 + 1e-6, 2]:
- tsne = TSNE(angle=angle)
+ tsne = TSNE(angle=angle, perplexity=1)
with pytest.raises(ValueError, match="'angle' must be between 0.0 - 1.0"):
tsne.fit_transform(np.array([[0.0], [1.0]]))
@@ -550,7 +552,7 @@ def test_angle_out_of_range_checks():
@pytest.mark.filterwarnings("ignore:The default learning rate in TSNE")
def test_pca_initialization_not_compatible_with_precomputed_kernel():
# Precomputed distance matrices cannot use PCA initialization.
- tsne = TSNE(metric="precomputed", init="pca")
+ tsne = TSNE(metric="precomputed", init="pca", perplexity=1)
with pytest.raises(
ValueError,
match='The parameter init="pca" cannot be used with metric="precomputed".',
@@ -560,7 +562,7 @@ def test_pca_initialization_not_compatible_with_precomputed_kernel():
def test_pca_initialization_not_compatible_with_sparse_input():
# Sparse input matrices cannot use PCA initialization.
- tsne = TSNE(init="pca", learning_rate=100.0)
+ tsne = TSNE(init="pca", learning_rate=100.0, perplexity=1)
with pytest.raises(TypeError, match="PCA initialization.*"):
tsne.fit_transform(sp.csr_matrix([[0, 5], [5, 0]]))
@@ -569,7 +571,7 @@ def test_pca_initialization_not_compatible_with_sparse_input():
@pytest.mark.filterwarnings("ignore:The default initialization in TSNE")
def test_n_components_range():
# barnes_hut method should only be used with n_components <= 3
- tsne = TSNE(n_components=4, method="barnes_hut")
+ tsne = TSNE(n_components=4, method="barnes_hut", perplexity=1)
with pytest.raises(ValueError, match="'n_components' should be .*"):
tsne.fit_transform(np.array([[0.0], [1.0]]))
@@ -736,7 +738,7 @@ def _run_answer_test(
def test_verbose():
# Verbose options write to stdout.
random_state = check_random_state(0)
- tsne = TSNE(verbose=2)
+ tsne = TSNE(verbose=2, perplexity=4)
X = random_state.randn(5, 2)
old_stdout = sys.stdout
@@ -760,7 +762,7 @@ def test_verbose():
def test_chebyshev_metric():
# t-SNE should allow metrics that cannot be squared (issue #3526).
random_state = check_random_state(0)
- tsne = TSNE(metric="chebyshev")
+ tsne = TSNE(metric="chebyshev", perplexity=4)
X = random_state.randn(5, 2)
tsne.fit_transform(X)
@@ -770,7 +772,7 @@ def test_chebyshev_metric():
def test_reduction_to_one_component():
# t-SNE should allow reduction to one component (issue #4154).
random_state = check_random_state(0)
- tsne = TSNE(n_components=1)
+ tsne = TSNE(n_components=1, perplexity=4)
X = random_state.randn(5, 2)
X_embedded = tsne.fit(X).embedding_
assert np.all(np.isfinite(X_embedded))
@@ -1042,7 +1044,7 @@ def test_bh_match_exact():
init="random",
random_state=0,
n_iter=251,
- perplexity=30.0,
+ perplexity=29.5,
angle=0,
)
# Kill the early_exaggeration
@@ -1141,7 +1143,7 @@ def test_tsne_init_futurewarning(init):
random_state = check_random_state(0)
X = random_state.randn(5, 2)
- kwargs = dict(learning_rate=200.0, init=init)
+ kwargs = dict(learning_rate=200.0, init=init, perplexity=4)
tsne = TSNE(**{k: v for k, v in kwargs.items() if v is not None})
if init is None:
@@ -1164,7 +1166,7 @@ def test_tsne_learning_rate_futurewarning(learning_rate):
random_state = check_random_state(0)
X = random_state.randn(5, 2)
- kwargs = dict(learning_rate=learning_rate, init="random")
+ kwargs = dict(learning_rate=learning_rate, init="random", perplexity=4)
tsne = TSNE(**{k: v for k, v in kwargs.items() if v is not None})
if learning_rate is None:
@@ -1182,7 +1184,7 @@ def test_tsne_negative_learning_rate():
random_state = check_random_state(0)
X = random_state.randn(5, 2)
with pytest.raises(ValueError, match="'learning_rate' must be.*"):
- TSNE(learning_rate=-50.0).fit_transform(X)
+ TSNE(learning_rate=-50.0, perplexity=4).fit_transform(X)
@pytest.mark.parametrize("method", ["exact", "barnes_hut"])
@@ -1194,7 +1196,7 @@ def test_tsne_n_jobs(method):
X_tr_ref = TSNE(
n_components=2,
method=method,
- perplexity=30.0,
+ perplexity=25.0,
angle=0,
n_jobs=1,
random_state=0,
@@ -1204,7 +1206,7 @@ def test_tsne_n_jobs(method):
X_tr = TSNE(
n_components=2,
method=method,
- perplexity=30.0,
+ perplexity=25.0,
angle=0,
n_jobs=2,
random_state=0,
@@ -1260,7 +1262,7 @@ def test_tsne_deprecation_square_distances():
n_components=2,
init="pca",
learning_rate="auto",
- perplexity=30.0,
+ perplexity=25.0,
angle=0,
n_jobs=1,
random_state=0,
@@ -1277,10 +1279,27 @@ def test_tsne_deprecation_square_distances():
n_components=2,
init="pca",
learning_rate="auto",
- perplexity=30.0,
+ perplexity=25.0,
angle=0,
n_jobs=1,
random_state=0,
)
X_trans_2 = tsne.fit_transform(X)
assert_allclose(X_trans_1, X_trans_2)
+
+
+@pytest.mark.parametrize("perplexity", (20, 30))
+def test_tsne_perplexity_validation(perplexity):
+ """Make sure that perplexity > n_samples results in a ValueError"""
+
+ random_state = check_random_state(0)
+ X = random_state.randn(20, 2)
+ est = TSNE(
+ learning_rate="auto",
+ init="pca",
+ perplexity=perplexity,
+ random_state=random_state,
+ )
+ msg = "perplexity must be less than n_samples"
+ with pytest.raises(ValueError, match=msg):
+ est.fit_transform(X)
diff --git a/sklearn/metrics/_classification.py b/sklearn/metrics/_classification.py
index b0a44e9d83d31..97f3abe2b884d 100644
--- a/sklearn/metrics/_classification.py
+++ b/sklearn/metrics/_classification.py
@@ -586,7 +586,7 @@ def multilabel_confusion_matrix(
def cohen_kappa_score(y1, y2, *, labels=None, weights=None, sample_weight=None):
- r"""Cohen's kappa: a statistic that measures inter-annotator agreement.
+ r"""Compute Cohen's kappa: a statistic that measures inter-annotator agreement.
This function computes Cohen's kappa [1]_, a score that expresses the level
of agreement between two annotators on a classification problem. It is
@@ -738,12 +738,16 @@ def jaccard_score(
Returns
-------
- score : float (if average is not None) or array of floats, shape =\
- [n_unique_labels]
+ score : float or ndarray of shape (n_unique_labels,), dtype=np.float64
+ The Jaccard score. When `average` is not `None`, a single scalar is
+ returned.
See Also
--------
- accuracy_score, f1_score, multilabel_confusion_matrix
+ accuracy_score : Function for calculating the accuracy score.
+ f1_score : Function for calculating the F1 score.
+ multilabel_confusion_matrix : Function for computing a confusion matrix\
+ for each class or sample.
Notes
-----
@@ -1235,7 +1239,10 @@ def fbeta_score(
See Also
--------
- precision_recall_fscore_support, multilabel_confusion_matrix
+ precision_recall_fscore_support : Compute the precision, recall, F-score,
+ and support.
+ multilabel_confusion_matrix : Compute a confusion matrix for each class or
+ sample.
Notes
-----
@@ -2374,22 +2381,23 @@ def log_loss(
Returns
-------
loss : float
+ Log loss, aka logistic loss or cross-entropy loss.
Notes
-----
The logarithm used is the natural logarithm (base-e).
+ References
+ ----------
+ C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer,
+ p. 209.
+
Examples
--------
>>> from sklearn.metrics import log_loss
>>> log_loss(["spam", "ham", "ham", "spam"],
... [[.1, .9], [.9, .1], [.8, .2], [.35, .65]])
0.21616...
-
- References
- ----------
- C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer,
- p. 209.
"""
y_pred = check_array(y_pred, ensure_2d=False)
check_consistent_length(y_pred, y_true, sample_weight)
@@ -2508,8 +2516,7 @@ def hinge_loss(y_true, pred_decision, *, labels=None, sample_weight=None):
.. [3] `L1 AND L2 Regularization for Multiclass Hinge Loss Models
by Robert C. Moore, John DeNero
- `_.
+ `_.
Examples
--------
@@ -2653,6 +2660,11 @@ def brier_score_loss(y_true, y_prob, *, sample_weight=None, pos_label=None):
score : float
Brier score loss.
+ References
+ ----------
+ .. [1] `Wikipedia entry for the Brier score
+ `_.
+
Examples
--------
>>> import numpy as np
@@ -2668,11 +2680,6 @@ def brier_score_loss(y_true, y_prob, *, sample_weight=None, pos_label=None):
0.037...
>>> brier_score_loss(y_true, np.array(y_prob) > 0.5)
0.0
-
- References
- ----------
- .. [1] `Wikipedia entry for the Brier score
- `_.
"""
y_true = column_or_1d(y_true)
y_prob = column_or_1d(y_prob)
diff --git a/sklearn/metrics/_dist_metrics.pyx b/sklearn/metrics/_dist_metrics.pyx
index d17be2c8cb73d..953e6378adcb2 100644
--- a/sklearn/metrics/_dist_metrics.pyx
+++ b/sklearn/metrics/_dist_metrics.pyx
@@ -1097,72 +1097,6 @@ cdef class HaversineDistance(DistanceMetric):
tmp = np.sin(0.5 * dist)
return tmp * tmp
-
-#------------------------------------------------------------
-# Yule Distance (boolean)
-# D(x, y) = 2 * ntf * nft / (ntt * nff + ntf * nft)
-# [This is not a true metric, so we will leave it out.]
-#
-#cdef class YuleDistance(DistanceMetric):
-# cdef inline DTYPE_t dist(self, const DTYPE_t* x1, const DTYPE_t* x2,
-# ITYPE_t size):
-# cdef int tf1, tf2, ntf = 0, nft = 0, ntt = 0, nff = 0
-# cdef np.intp_t j
-# for j in range(size):
-# tf1 = x1[j] != 0
-# tf2 = x2[j] != 0
-# ntt += tf1 and tf2
-# ntf += tf1 and (tf2 == 0)
-# nft += (tf1 == 0) and tf2
-# nff = size - ntt - ntf - nft
-# return (2.0 * ntf * nft) / (ntt * nff + ntf * nft)
-
-
-#------------------------------------------------------------
-# Cosine Distance
-# D(x, y) = dot(x, y) / (|x| * |y|)
-# [This is not a true metric, so we will leave it out.]
-#
-#cdef class CosineDistance(DistanceMetric):
-# cdef inline DTYPE_t dist(self, const DTYPE_t* x1, const DTYPE_t* x2,
-# ITYPE_t size):
-# cdef DTYPE_t d = 0, norm1 = 0, norm2 = 0
-# cdef np.intp_t j
-# for j in range(size):
-# d += x1[j] * x2[j]
-# norm1 += x1[j] * x1[j]
-# norm2 += x2[j] * x2[j]
-# return 1.0 - d / sqrt(norm1 * norm2)
-
-
-#------------------------------------------------------------
-# Correlation Distance
-# D(x, y) = dot((x - mx), (y - my)) / (|x - mx| * |y - my|)
-# [This is not a true metric, so we will leave it out.]
-#
-#cdef class CorrelationDistance(DistanceMetric):
-# cdef inline DTYPE_t dist(self, const DTYPE_t* x1, const DTYPE_t* x2,
-# ITYPE_t size):
-# cdef DTYPE_t mu1 = 0, mu2 = 0, x1nrm = 0, x2nrm = 0, x1Tx2 = 0
-# cdef DTYPE_t tmp1, tmp2
-#
-# cdef np.intp_t i
-# for i in range(size):
-# mu1 += x1[i]
-# mu2 += x2[i]
-# mu1 /= size
-# mu2 /= size
-#
-# for i in range(size):
-# tmp1 = x1[i] - mu1
-# tmp2 = x2[i] - mu2
-# x1nrm += tmp1 * tmp1
-# x2nrm += tmp2 * tmp2
-# x1Tx2 += tmp1 * tmp2
-#
-# return (1. - x1Tx2) / sqrt(x1nrm * x2nrm)
-
-
#------------------------------------------------------------
# User-defined distance
#
diff --git a/sklearn/metrics/_pairwise_distances_reduction.pyx b/sklearn/metrics/_pairwise_distances_reduction.pyx
index a37bdb7ab0e8d..ff0d9947dfab8 100644
--- a/sklearn/metrics/_pairwise_distances_reduction.pyx
+++ b/sklearn/metrics/_pairwise_distances_reduction.pyx
@@ -245,9 +245,16 @@ cdef class PairwiseDistancesReduction:
True if the PairwiseDistancesReduction can be used, else False.
"""
# TODO: support sparse arrays and 32 bits
+ c_contiguity = (
+ hasattr(X, "flags")
+ and X.flags.c_contiguous
+ and hasattr(Y, "flags")
+ and Y.flags.c_contiguous
+ )
return (get_config().get("enable_cython_pairwise_dist", True) and
not issparse(X) and X.dtype == np.float64 and
not issparse(Y) and Y.dtype == np.float64 and
+ c_contiguity and
metric in cls.valid_metrics())
def __init__(
diff --git a/sklearn/metrics/_plot/roc_curve.py b/sklearn/metrics/_plot/roc_curve.py
index a56cd3755b8d6..49953d7032c71 100644
--- a/sklearn/metrics/_plot/roc_curve.py
+++ b/sklearn/metrics/_plot/roc_curve.py
@@ -352,8 +352,8 @@ def from_predictions(
@deprecated(
"Function :func:`plot_roc_curve` is deprecated in 1.0 and will be "
"removed in 1.2. Use one of the class methods: "
- ":meth:`sklearn.metric.RocCurveDisplay.from_predictions` or "
- ":meth:`sklearn.metric.RocCurveDisplay.from_estimator`."
+ ":meth:`sklearn.metrics.RocCurveDisplay.from_predictions` or "
+ ":meth:`sklearn.metrics.RocCurveDisplay.from_estimator`."
)
def plot_roc_curve(
estimator,
diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index 4e88bd5edc888..81e201fe5a349 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -55,14 +55,15 @@ def auc(x, y):
Parameters
----------
x : ndarray of shape (n,)
- x coordinates. These must be either monotonic increasing or monotonic
+ X coordinates. These must be either monotonic increasing or monotonic
decreasing.
y : ndarray of shape, (n,)
- y coordinates.
+ Y coordinates.
Returns
-------
auc : float
+ Area Under the Curve.
See Also
--------
@@ -371,7 +372,7 @@ def roc_auc_score(
multi_class="raise",
labels=None,
):
- """Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC)
+ """Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) \
from prediction scores.
Note: this implementation can be used with binary, multiclass and
@@ -470,6 +471,16 @@ class scores must correspond to the order of ``labels``,
Returns
-------
auc : float
+ Area Under the Curve score.
+
+ See Also
+ --------
+ average_precision_score : Area under the precision-recall curve.
+ roc_curve : Compute Receiver operating characteristic (ROC) curve.
+ RocCurveDisplay.from_estimator : Plot Receiver Operating Characteristic
+ (ROC) curve given an estimator and some data.
+ RocCurveDisplay.from_predictions : Plot Receiver Operating Characteristic
+ (ROC) curve given the true and predicted values.
References
----------
@@ -492,15 +503,6 @@ class scores must correspond to the order of ``labels``,
Machine Learning, 45(2), 171-186.
`_
- See Also
- --------
- average_precision_score : Area under the precision-recall curve.
- roc_curve : Compute Receiver operating characteristic (ROC) curve.
- RocCurveDisplay.from_estimator : Plot Receiver Operating Characteristic
- (ROC) curve given an estimator and some data.
- RocCurveDisplay.from_predictions : Plot Receiver Operating Characteristic
- (ROC) curve given the true and predicted values.
-
Examples
--------
Binary case:
@@ -1057,6 +1059,7 @@ def label_ranking_average_precision_score(y_true, y_score, *, sample_weight=None
Returns
-------
score : float
+ Ranking-based average precision score.
Examples
--------
@@ -1066,7 +1069,6 @@ def label_ranking_average_precision_score(y_true, y_score, *, sample_weight=None
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416...
-
"""
check_consistent_length(y_true, y_score, sample_weight)
y_true = check_array(y_true, ensure_2d=False)
@@ -1153,8 +1155,8 @@ def coverage_error(y_true, y_score, *, sample_weight=None):
handbook (pp. 667-685). Springer US.
"""
- y_true = check_array(y_true, ensure_2d=False)
- y_score = check_array(y_score, ensure_2d=False)
+ y_true = check_array(y_true, ensure_2d=True)
+ y_score = check_array(y_score, ensure_2d=True)
check_consistent_length(y_true, y_score, sample_weight)
y_type = type_of_target(y_true, input_name="y_true")
@@ -1617,19 +1619,19 @@ def ndcg_score(y_true, y_score, *, k=None, sample_weight=None, ignore_ties=False
>>> # the normalization takes k into account so a perfect answer
>>> # would still get 1.0
>>> ndcg_score(true_relevance, true_relevance, k=4)
- 1.0
+ 1.0...
>>> # now we have some ties in our prediction
>>> scores = np.asarray([[1, 0, 0, 0, 1]])
>>> # by default ties are averaged, so here we get the average (normalized)
>>> # true relevance of our top predictions: (10 / 10 + 5 / 10) / 2 = .75
>>> ndcg_score(true_relevance, scores, k=1)
- 0.75
+ 0.75...
>>> # we can choose to ignore ties for faster results, but only
>>> # if we know there aren't ties in our scores, otherwise we get
>>> # wrong results:
>>> ndcg_score(true_relevance,
... scores, k=1, ignore_ties=True)
- 0.5
+ 0.5...
"""
y_true = check_array(y_true, ensure_2d=False)
y_score = check_array(y_score, ensure_2d=False)
diff --git a/sklearn/metrics/_regression.py b/sklearn/metrics/_regression.py
index 57986692fb896..9d4315470bc97 100644
--- a/sklearn/metrics/_regression.py
+++ b/sklearn/metrics/_regression.py
@@ -1223,7 +1223,7 @@ def d2_tweedie_score(y_true, y_pred, *, sample_weight=None, power=0):
----------
.. [1] Eq. (3.11) of Hastie, Trevor J., Robert Tibshirani and Martin J.
Wainwright. "Statistical Learning with Sparsity: The Lasso and
- Generalizations." (2015). https://trevorhastie.github.io
+ Generalizations." (2015). https://hastie.su.domains/StatLearnSparsity/
Examples
--------
@@ -1326,7 +1326,7 @@ def d2_pinball_score(
`_
.. [2] Eq. (3.11) of Hastie, Trevor J., Robert Tibshirani and Martin J.
Wainwright. "Statistical Learning with Sparsity: The Lasso and
- Generalizations." (2015). https://trevorhastie.github.io
+ Generalizations." (2015). https://hastie.su.domains/StatLearnSparsity/
Examples
--------
@@ -1464,7 +1464,7 @@ def d2_absolute_error_score(
----------
.. [1] Eq. (3.11) of Hastie, Trevor J., Robert Tibshirani and Martin J.
Wainwright. "Statistical Learning with Sparsity: The Lasso and
- Generalizations." (2015). https://trevorhastie.github.io
+ Generalizations." (2015). https://hastie.su.domains/StatLearnSparsity/
Examples
--------
diff --git a/sklearn/metrics/cluster/_supervised.py b/sklearn/metrics/cluster/_supervised.py
index a6a66884b70b2..3add417716089 100644
--- a/sklearn/metrics/cluster/_supervised.py
+++ b/sklearn/metrics/cluster/_supervised.py
@@ -159,7 +159,7 @@ def contingency_matrix(
def pair_confusion_matrix(labels_true, labels_pred):
- """Pair confusion matrix arising from two clusterings.
+ """Pair confusion matrix arising from two clusterings [1]_.
The pair confusion matrix :math:`C` computes a 2 by 2 similarity matrix
between two clusterings by considering all pairs of samples and counting
@@ -188,9 +188,15 @@ def pair_confusion_matrix(labels_true, labels_pred):
See Also
--------
- rand_score: Rand Score
- adjusted_rand_score: Adjusted Rand Score
- adjusted_mutual_info_score: Adjusted Mutual Information
+ rand_score: Rand Score.
+ adjusted_rand_score: Adjusted Rand Score.
+ adjusted_mutual_info_score: Adjusted Mutual Information.
+
+ References
+ ----------
+ .. [1] :doi:`Hubert, L., Arabie, P. "Comparing partitions."
+ Journal of Classification 2, 193–218 (1985).
+ <10.1007/BF01908075>`
Examples
--------
@@ -211,12 +217,6 @@ def pair_confusion_matrix(labels_true, labels_pred):
[0, 2]]...
Note that the matrix is not symmetric.
-
- References
- ----------
- .. L. Hubert and P. Arabie, Comparing Partitions, Journal of
- Classification 1985
- https://link.springer.com/article/10.1007%2FBF01908075
"""
labels_true, labels_pred = check_clusterings(labels_true, labels_pred)
n_samples = np.int64(labels_true.shape[0])
@@ -429,10 +429,10 @@ def homogeneity_completeness_v_measure(labels_true, labels_pred, *, beta=1.0):
Parameters
----------
labels_true : int array, shape = [n_samples]
- ground truth class labels to be used as a reference
+ Ground truth class labels to be used as a reference.
labels_pred : array-like of shape (n_samples,)
- cluster labels to evaluate
+ Gluster labels to evaluate.
beta : float, default=1.0
Ratio of weight attributed to ``homogeneity`` vs ``completeness``.
@@ -443,19 +443,19 @@ def homogeneity_completeness_v_measure(labels_true, labels_pred, *, beta=1.0):
Returns
-------
homogeneity : float
- score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling
+ Score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling.
completeness : float
- score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling
+ Score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling.
v_measure : float
- harmonic mean of the first two
+ Harmonic mean of the first two.
See Also
--------
- homogeneity_score
- completeness_score
- v_measure_score
+ homogeneity_score : Homogeneity metric of cluster labeling.
+ completeness_score : Completeness metric of cluster labeling.
+ v_measure_score : V-Measure (NMI with arithmetic mean option).
"""
labels_true, labels_pred = check_clusterings(labels_true, labels_pred)
@@ -689,7 +689,7 @@ def v_measure_score(labels_true, labels_pred, *, beta=1.0):
1.0
Labelings that assign all classes members to the same clusters
- are complete be not homogeneous, hence penalized::
+ are complete but not homogeneous, hence penalized::
>>> print("%.6f" % v_measure_score([0, 0, 1, 2], [0, 0, 1, 1]))
0.8...
@@ -697,7 +697,7 @@ def v_measure_score(labels_true, labels_pred, *, beta=1.0):
0.66...
Labelings that have pure clusters with members coming from the same
- classes are homogeneous but un-necessary splits harms completeness
+ classes are homogeneous but un-necessary splits harm completeness
and thus penalize V-measure as well::
>>> print("%.6f" % v_measure_score([0, 0, 1, 1], [0, 0, 1, 2]))
@@ -768,14 +768,14 @@ def mutual_info_score(labels_true, labels_pred, *, contingency=None):
Mutual information, a non-negative value, measured in nats using the
natural logarithm.
- Notes
- -----
- The logarithm used is the natural logarithm (base-e).
-
See Also
--------
adjusted_mutual_info_score : Adjusted against chance Mutual Information.
normalized_mutual_info_score : Normalized Mutual Information.
+
+ Notes
+ -----
+ The logarithm used is the natural logarithm (base-e).
"""
if contingency is None:
labels_true, labels_pred = check_clusterings(labels_true, labels_pred)
diff --git a/sklearn/metrics/pairwise.py b/sklearn/metrics/pairwise.py
index 33b2a9901902b..bcf01bc2925a3 100644
--- a/sklearn/metrics/pairwise.py
+++ b/sklearn/metrics/pairwise.py
@@ -135,7 +135,6 @@ def check_pairwise_arrays(
safe_Y : {array-like, sparse matrix} of shape (n_samples_Y, n_features)
An array equal to Y if Y was not None, guaranteed to be a numpy array.
If Y was None, safe_Y will be a pointer to X.
-
"""
X, Y, dtype_float = _return_float_dtype(X, Y)
@@ -211,7 +210,6 @@ def check_paired_arrays(X, Y):
safe_Y : {array-like, sparse matrix} of shape (n_samples_Y, n_features)
An array equal to Y if Y was not None, guaranteed to be a numpy array.
If Y was None, safe_Y will be a pointer to X.
-
"""
X, Y = check_pairwise_arrays(X, Y)
if X.shape != Y.shape:
@@ -654,13 +652,14 @@ def pairwise_distances_argmin_min(
Y[argmin[i], :] is the row in Y that is closest to X[i, :].
distances : ndarray
- distances[i] is the distance between the i-th row in X and the
- argmin[i]-th row in Y.
+ The array of minimum distances. `distances[i]` is the distance between
+ the i-th row in X and the argmin[i]-th row in Y.
See Also
--------
- sklearn.metrics.pairwise_distances
- sklearn.metrics.pairwise_distances_argmin
+ pairwise_distances : Distances between every pair of samples of X and Y.
+ pairwise_distances_argmin : Same as `pairwise_distances_argmin_min` but only
+ returns the argmins.
"""
X, Y = check_pairwise_arrays(X, Y)
@@ -767,8 +766,9 @@ def pairwise_distances_argmin(X, Y, *, axis=1, metric="euclidean", metric_kwargs
See Also
--------
- sklearn.metrics.pairwise_distances
- sklearn.metrics.pairwise_distances_argmin_min
+ pairwise_distances : Distances between every pair of samples of X and Y.
+ pairwise_distances_argmin_min : Same as `pairwise_distances_argmin` but also
+ returns the distances.
"""
if metric_kwargs is None:
metric_kwargs = {}
@@ -973,10 +973,11 @@ def cosine_distances(X, Y=None):
Returns
-------
distance matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ Returns the cosine distance between samples in X and Y.
See Also
--------
- cosine_similarity
+ cosine_similarity : Compute cosine similarity between samples in X and Y.
scipy.spatial.distance.cosine : Dense matrices only.
"""
# 1.0 - cosine_similarity(X, Y) without copy
@@ -1016,19 +1017,35 @@ def paired_euclidean_distances(X, Y):
def paired_manhattan_distances(X, Y):
- """Compute the L1 distances between the vectors in X and Y.
+ """Compute the paired L1 distances between X and Y.
+
+ Distances are calculated between (X[0], Y[0]), (X[1], Y[1]), ...,
+ (X[n_samples], Y[n_samples]).
Read more in the :ref:`User Guide `.
Parameters
----------
X : array-like of shape (n_samples, n_features)
+ An array-like where each row is a sample and each column is a feature.
Y : array-like of shape (n_samples, n_features)
+ An array-like where each row is a sample and each column is a feature.
Returns
-------
distances : ndarray of shape (n_samples,)
+ L1 paired distances between the row vectors of `X`
+ and the row vectors of `Y`.
+
+ Examples
+ --------
+ >>> from sklearn.metrics.pairwise import paired_manhattan_distances
+ >>> import numpy as np
+ >>> X = np.array([[1, 1, 0], [0, 1, 0], [0, 0, 1]])
+ >>> Y = np.array([[0, 1, 0], [0, 0, 1], [0, 0, 0]])
+ >>> paired_manhattan_distances(X, Y)
+ array([1., 2., 1.])
"""
X, Y = check_paired_arrays(X, Y)
diff = X - Y
@@ -1173,28 +1190,33 @@ def linear_kernel(X, Y=None, dense_output=True):
def polynomial_kernel(X, Y=None, degree=3, gamma=None, coef0=1):
"""
- Compute the polynomial kernel between X and Y::
+ Compute the polynomial kernel between X and Y.
- K(X, Y) = (gamma + coef0)^degree
+ :math:`K(X, Y) = (gamma + coef0)^degree`
Read more in the :ref:`User Guide `.
Parameters
----------
X : ndarray of shape (n_samples_X, n_features)
+ A feature array.
Y : ndarray of shape (n_samples_Y, n_features), default=None
+ An optional second feature array. If `None`, uses `Y=X`.
degree : int, default=3
+ Kernel degree.
gamma : float, default=None
- If None, defaults to 1.0 / n_features.
+ Coefficient of the vector inner product. If None, defaults to 1.0 / n_features.
coef0 : float, default=1
+ Constant offset added to scaled inner product.
Returns
-------
Gram matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ The polynomial kernel.
"""
X, Y = check_pairwise_arrays(X, Y)
if gamma is None:
@@ -1208,8 +1230,7 @@ def polynomial_kernel(X, Y=None, degree=3, gamma=None, coef0=1):
def sigmoid_kernel(X, Y=None, gamma=None, coef0=1):
- """
- Compute the sigmoid kernel between X and Y::
+ """Compute the sigmoid kernel between X and Y.
K(X, Y) = tanh(gamma + coef0)
@@ -1218,18 +1239,21 @@ def sigmoid_kernel(X, Y=None, gamma=None, coef0=1):
Parameters
----------
X : ndarray of shape (n_samples_X, n_features)
+ A feature array.
Y : ndarray of shape (n_samples_Y, n_features), default=None
- If `None`, uses `Y=X`.
+ An optional second feature array. If `None`, uses `Y=X`.
gamma : float, default=None
- If None, defaults to 1.0 / n_features.
+ Coefficient of the vector inner product. If None, defaults to 1.0 / n_features.
coef0 : float, default=1
+ Constant offset added to scaled inner product.
Returns
-------
Gram matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ Sigmoid kernel between two arrays.
"""
X, Y = check_pairwise_arrays(X, Y)
if gamma is None:
@@ -1243,8 +1267,7 @@ def sigmoid_kernel(X, Y=None, gamma=None, coef0=1):
def rbf_kernel(X, Y=None, gamma=None):
- """
- Compute the rbf (gaussian) kernel between X and Y::
+ """Compute the rbf (gaussian) kernel between X and Y.
K(x, y) = exp(-gamma ||x-y||^2)
@@ -1255,9 +1278,10 @@ def rbf_kernel(X, Y=None, gamma=None):
Parameters
----------
X : ndarray of shape (n_samples_X, n_features)
+ A feature array.
Y : ndarray of shape (n_samples_Y, n_features), default=None
- If `None`, uses `Y=X`.
+ An optional second feature array. If `None`, uses `Y=X`.
gamma : float, default=None
If None, defaults to 1.0 / n_features.
@@ -1265,6 +1289,7 @@ def rbf_kernel(X, Y=None, gamma=None):
Returns
-------
kernel_matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ The RBF kernel.
"""
X, Y = check_pairwise_arrays(X, Y)
if gamma is None:
@@ -1345,6 +1370,7 @@ def cosine_similarity(X, Y=None, dense_output=True):
Returns
-------
kernel matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ Returns the cosine similarity between samples in X and Y.
"""
# to avoid recursive import
@@ -1362,8 +1388,7 @@ def cosine_similarity(X, Y=None, dense_output=True):
def additive_chi2_kernel(X, Y=None):
- """Computes the additive chi-squared kernel between observations in X and
- Y.
+ """Compute the additive chi-squared kernel between observations in X and Y.
The chi-squared kernel is computed between each pair of rows in X and Y. X
and Y have to be non-negative. This kernel is most commonly applied to
@@ -1377,22 +1402,18 @@ def additive_chi2_kernel(X, Y=None):
Read more in the :ref:`User Guide `.
- Notes
- -----
- As the negative of a distance, this kernel is only conditionally positive
- definite.
-
-
Parameters
----------
X : array-like of shape (n_samples_X, n_features)
+ A feature array.
Y : ndarray of shape (n_samples_Y, n_features), default=None
- If `None`, uses `Y=X`.
+ An optional second feature array. If `None`, uses `Y=X`.
Returns
-------
kernel_matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ The kernel matrix.
See Also
--------
@@ -1401,13 +1422,18 @@ def additive_chi2_kernel(X, Y=None):
sklearn.kernel_approximation.AdditiveChi2Sampler : A Fourier approximation
to this kernel.
+ Notes
+ -----
+ As the negative of a distance, this kernel is only conditionally positive
+ definite.
+
References
----------
* Zhang, J. and Marszalek, M. and Lazebnik, S. and Schmid, C.
Local features and kernels for classification of texture and object
categories: A comprehensive study
International Journal of Computer Vision 2007
- https://research.microsoft.com/en-us/um/people/manik/projects/trade-off/papers/ZhangIJCV06.pdf
+ https://hal.archives-ouvertes.fr/hal-00171412/document
"""
if issparse(X) or issparse(Y):
raise ValueError("additive_chi2 does not support sparse matrices.")
@@ -1423,7 +1449,7 @@ def additive_chi2_kernel(X, Y=None):
def chi2_kernel(X, Y=None, gamma=1.0):
- """Computes the exponential chi-squared kernel X and Y.
+ """Compute the exponential chi-squared kernel between X and Y.
The chi-squared kernel is computed between each pair of rows in X and Y. X
and Y have to be non-negative. This kernel is most commonly applied to
@@ -1440,15 +1466,18 @@ def chi2_kernel(X, Y=None, gamma=1.0):
Parameters
----------
X : array-like of shape (n_samples_X, n_features)
+ A feature array.
Y : ndarray of shape (n_samples_Y, n_features), default=None
+ An optional second feature array. If `None`, uses `Y=X`.
- gamma : float, default=1.
+ gamma : float, default=1
Scaling parameter of the chi2 kernel.
Returns
-------
kernel_matrix : ndarray of shape (n_samples_X, n_samples_Y)
+ The kernel matrix.
See Also
--------
@@ -1462,7 +1491,7 @@ def chi2_kernel(X, Y=None, gamma=1.0):
Local features and kernels for classification of texture and object
categories: A comprehensive study
International Journal of Computer Vision 2007
- https://research.microsoft.com/en-us/um/people/manik/projects/trade-off/papers/ZhangIJCV06.pdf
+ https://hal.archives-ouvertes.fr/hal-00171412/document
"""
K = additive_chi2_kernel(X, Y)
K *= gamma
@@ -1509,6 +1538,10 @@ def distance_metrics():
Read more in the :ref:`User Guide `.
+ Returns
+ -------
+ distance_metrics : dict
+ Returns valid metrics for pairwise_distances.
"""
return PAIRWISE_DISTANCE_FUNCTIONS
@@ -1862,7 +1895,7 @@ def pairwise_distances(
valid scipy.spatial.distance metrics), the scikit-learn implementation
will be used, which is faster and has support for sparse matrices (except
for 'cityblock'). For a verbose description of the metrics from
- scikit-learn, see the __doc__ of the sklearn.pairwise.distance_metrics
+ scikit-learn, see :func:`sklearn.metrics.pairwise.distance_metrics`
function.
Read more in the :ref:`User Guide `.
@@ -2041,6 +2074,11 @@ def kernel_metrics():
=============== ========================================
Read more in the :ref:`User Guide `.
+
+ Returns
+ -------
+ kernal_metrics : dict
+ Returns valid metrics for pairwise_kernels.
"""
return PAIRWISE_KERNEL_FUNCTIONS
@@ -2082,8 +2120,7 @@ def pairwise_kernels(
Parameters
----------
- X : ndarray of shape (n_samples_X, n_samples_X) or \
- (n_samples_X, n_features)
+ X : ndarray of shape (n_samples_X, n_samples_X) or (n_samples_X, n_features)
Array of pairwise kernels between samples, or a feature array.
The shape of the array should be (n_samples_X, n_samples_X) if
metric == "precomputed" and (n_samples_X, n_features) otherwise.
@@ -2121,8 +2158,7 @@ def pairwise_kernels(
Returns
-------
- K : ndarray of shape (n_samples_X, n_samples_X) or \
- (n_samples_X, n_samples_Y)
+ K : ndarray of shape (n_samples_X, n_samples_X) or (n_samples_X, n_samples_Y)
A kernel matrix K such that K_{i, j} is the kernel between the
ith and jth vectors of the given matrix X, if Y is None.
If Y is not None, then K_{i, j} is the kernel between the ith array
@@ -2131,7 +2167,6 @@ def pairwise_kernels(
Notes
-----
If metric is 'precomputed', Y is ignored and X is returned.
-
"""
# import GPKernel locally to prevent circular imports
from ..gaussian_process.kernels import Kernel as GPKernel
diff --git a/sklearn/metrics/tests/test_common.py b/sklearn/metrics/tests/test_common.py
index 1e627f9f86676..c0d6d351b8c3e 100644
--- a/sklearn/metrics/tests/test_common.py
+++ b/sklearn/metrics/tests/test_common.py
@@ -910,6 +910,10 @@ def test_thresholded_invariance_string_vs_numbers_labels(name):
)
@pytest.mark.parametrize("y_true, y_score", invalids_nan_inf)
def test_regression_thresholded_inf_nan_input(metric, y_true, y_score):
+ # Reshape since coverage_error only accepts 2D arrays.
+ if metric == coverage_error:
+ y_true = [y_true]
+ y_score = [y_score]
with pytest.raises(ValueError, match=r"contains (NaN|infinity)"):
metric(y_true, y_score)
diff --git a/sklearn/metrics/tests/test_pairwise.py b/sklearn/metrics/tests/test_pairwise.py
index f14c558d5a3c1..31964e2d182dd 100644
--- a/sklearn/metrics/tests/test_pairwise.py
+++ b/sklearn/metrics/tests/test_pairwise.py
@@ -534,6 +534,14 @@ def test_pairwise_distances_argmin_min(dtype):
assert_array_equal(argmin_0, argmin_1)
+ # F-contiguous arrays must be supported and must return identical results.
+ argmin_C_contiguous = pairwise_distances_argmin(X, Y)
+ argmin_F_contiguous = pairwise_distances_argmin(
+ np.asfortranarray(X), np.asfortranarray(Y)
+ )
+
+ assert_array_equal(argmin_C_contiguous, argmin_F_contiguous)
+
def _reduce_func(dist, start):
return dist[:, :100]
diff --git a/sklearn/metrics/tests/test_pairwise_distances_reduction.py b/sklearn/metrics/tests/test_pairwise_distances_reduction.py
index 192f7ef43a6c6..c6a71596e3472 100644
--- a/sklearn/metrics/tests/test_pairwise_distances_reduction.py
+++ b/sklearn/metrics/tests/test_pairwise_distances_reduction.py
@@ -121,6 +121,9 @@ def test_pairwise_distances_reduction_is_usable_for():
assert not PairwiseDistancesReduction.is_usable_for(csr_matrix(X), Y, metric)
assert not PairwiseDistancesReduction.is_usable_for(X, csr_matrix(Y), metric)
+ # F-ordered arrays are not supported
+ assert not PairwiseDistancesReduction.is_usable_for(np.asfortranarray(X), Y, metric)
+
def test_argkmin_factory_method_wrong_usages():
rng = np.random.RandomState(1)
@@ -510,9 +513,6 @@ def test_pairwise_distances_radius_neighbors(
neigh_indices_ref.append(ind)
neigh_distances_ref.append(dist)
- neigh_indices_ref = np.array(neigh_indices_ref)
- neigh_distances_ref = np.array(neigh_distances_ref)
-
neigh_distances, neigh_indices = PairwiseDistancesRadiusNeighborhood.compute(
X,
Y,
diff --git a/sklearn/metrics/tests/test_ranking.py b/sklearn/metrics/tests/test_ranking.py
index 7d2338337b83d..c27d0f326fc0b 100644
--- a/sklearn/metrics/tests/test_ranking.py
+++ b/sklearn/metrics/tests/test_ranking.py
@@ -1569,6 +1569,21 @@ def test_coverage_tie_handling():
assert_almost_equal(coverage_error([[1, 1, 1]], [[0.25, 0.5, 0.5]]), 3)
+@pytest.mark.parametrize(
+ "y_true, y_score",
+ [
+ ([1, 0, 1], [0.25, 0.5, 0.5]),
+ ([1, 0, 1], [[0.25, 0.5, 0.5]]),
+ ([[1, 0, 1]], [0.25, 0.5, 0.5]),
+ ],
+)
+def test_coverage_1d_error_message(y_true, y_score):
+ # Non-regression test for:
+ # https://github.com/scikit-learn/scikit-learn/issues/23368
+ with pytest.raises(ValueError, match=r"Expected 2D array, got 1D array instead"):
+ coverage_error(y_true, y_score)
+
+
def test_label_ranking_loss():
assert_almost_equal(label_ranking_loss([[0, 1]], [[0.25, 0.75]]), 0)
assert_almost_equal(label_ranking_loss([[0, 1]], [[0.75, 0.25]]), 1)
diff --git a/sklearn/metrics/tests/test_regression.py b/sklearn/metrics/tests/test_regression.py
index 090bc64bf0fe4..b51012d6c1f1b 100644
--- a/sklearn/metrics/tests/test_regression.py
+++ b/sklearn/metrics/tests/test_regression.py
@@ -613,3 +613,15 @@ def test_dummy_quantile_parameter_tuning():
).fit(X, y)
assert grid_search.best_params_["quantile"] == pytest.approx(alpha)
+
+
+def test_pinball_loss_relation_with_mae():
+ # Test that mean_pinball loss with alpha=0.5 if half of mean absolute error
+ rng = np.random.RandomState(714)
+ n = 100
+ y_true = rng.normal(size=n)
+ y_pred = y_true.copy() + rng.uniform(n)
+ assert (
+ mean_absolute_error(y_true, y_pred)
+ == mean_pinball_loss(y_true, y_pred, alpha=0.5) * 2
+ )
diff --git a/sklearn/model_selection/_search.py b/sklearn/model_selection/_search.py
index 5ceb71569b932..4e2a0bd1c5207 100644
--- a/sklearn/model_selection/_search.py
+++ b/sklearn/model_selection/_search.py
@@ -1101,6 +1101,10 @@ class GridSearchCV(BaseSearchCV):
See ``scoring`` parameter to know more about multiple metric
evaluation.
+ See :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_digits.py`
+ to see how to design a custom selection strategy using a callable
+ via `refit`.
+
.. versionchanged:: 0.20
Support for callable added.
diff --git a/sklearn/model_selection/_split.py b/sklearn/model_selection/_split.py
index d2a0b5e1fc329..3be828b4c6114 100644
--- a/sklearn/model_selection/_split.py
+++ b/sklearn/model_selection/_split.py
@@ -422,7 +422,7 @@ class KFold(_BaseKFold):
See Also
--------
- StratifiedKFold : Takes group information into account to avoid building
+ StratifiedKFold : Takes class information into account to avoid building
folds with imbalanced class distributions (for binary or multiclass
classification tasks).
@@ -453,8 +453,8 @@ def _iter_test_indices(self, X, y=None, groups=None):
class GroupKFold(_BaseKFold):
"""K-fold iterator variant with non-overlapping groups.
- The same group will not appear in two different folds (the number of
- distinct groups has to be at least equal to the number of folds).
+ Each group will appear exactly once in the test set across all folds (the
+ number of distinct groups has to be at least equal to the number of folds).
The folds are approximately balanced in the sense that the number of
distinct groups is approximately the same in each fold.
@@ -504,6 +504,10 @@ class GroupKFold(_BaseKFold):
--------
LeaveOneGroupOut : For splitting the data according to explicit
domain-specific stratification of the dataset.
+
+ StratifiedKFold : Takes class information into account to avoid building
+ folds with imbalanced class proportions (for binary or multiclass
+ classification tasks).
"""
def __init__(self, n_splits=5):
@@ -759,10 +763,11 @@ class StratifiedGroupKFold(_BaseKFold):
return stratified folds with non-overlapping groups. The folds are made by
preserving the percentage of samples for each class.
- The same group will not appear in two different folds (the number of
- distinct groups has to be at least equal to the number of folds).
+ Each group will appear exactly once in the test set across all folds (the
+ number of distinct groups has to be at least equal to the number of folds).
- The difference between GroupKFold and StratifiedGroupKFold is that
+ The difference between :class:`~sklearn.model_selection.GroupKFold`
+ and :class:`~sklearn.model_selection.StratifiedGroupKFold` is that
the former attempts to create balanced folds such that the number of
distinct groups is approximately the same in each fold, whereas
StratifiedGroupKFold attempts to create folds which preserve the
@@ -1147,6 +1152,10 @@ class LeaveOneGroupOut(BaseCrossValidator):
[[1 2]
[3 4]] [[5 6]
[7 8]] [1 2] [1 2]
+
+ See also
+ --------
+ GroupKFold: K-fold iterator variant with non-overlapping groups.
"""
def _iter_test_masks(self, X, y, groups):
@@ -1799,6 +1808,12 @@ class GroupShuffleSplit(ShuffleSplit):
... print("TRAIN:", train_idx, "TEST:", test_idx)
TRAIN: [2 3 4 5 6 7] TEST: [0 1]
TRAIN: [0 1 5 6 7] TEST: [2 3 4]
+
+ See Also
+ --------
+ ShuffleSplit : Shuffles samples to create independent test/train sets.
+
+ LeavePGroupsOut : Train set leaves out all possible subsets of `p` groups.
"""
def __init__(
diff --git a/sklearn/model_selection/_validation.py b/sklearn/model_selection/_validation.py
index d83fca63da48c..94bbf532f5a59 100644
--- a/sklearn/model_selection/_validation.py
+++ b/sklearn/model_selection/_validation.py
@@ -1180,7 +1180,7 @@ def permutation_test_score(
scoring=None,
fit_params=None,
):
- """Evaluate the significance of a cross-validated score with permutations
+ """Evaluate the significance of a cross-validated score with permutations.
Permutes targets to generate 'randomized data' and compute the empirical
p-value against the null hypothesis that features and targets are
@@ -1218,12 +1218,6 @@ def permutation_test_score(
cross-validator uses them for grouping the samples while splitting
the dataset into train/test set.
- scoring : str or callable, default=None
- A single str (see :ref:`scoring_parameter`) or a callable
- (see :ref:`scoring`) to evaluate the predictions on the test set.
-
- If `None` the estimator's score method is used.
-
cv : int, cross-validation generator or an iterable, default=None
Determines the cross-validation splitting strategy.
Possible inputs for cv are:
@@ -1261,6 +1255,12 @@ def permutation_test_score(
verbose : int, default=0
The verbosity level.
+ scoring : str or callable, default=None
+ A single str (see :ref:`scoring_parameter`) or a callable
+ (see :ref:`scoring`) to evaluate the predictions on the test set.
+
+ If `None` the estimator's score method is used.
+
fit_params : dict, default=None
Parameters to pass to the fit method of the estimator.
@@ -1292,7 +1292,6 @@ def permutation_test_score(
Performance
`_. The
Journal of Machine Learning Research (2010) vol. 11
-
"""
X, y, groups = indexable(X, y, groups)
@@ -1805,11 +1804,6 @@ def validation_curve(
verbose : int, default=0
Controls the verbosity: the higher, the more messages.
- fit_params : dict, default=None
- Parameters to pass to the fit method of the estimator.
-
- .. versionadded:: 0.24
-
error_score : 'raise' or numeric, default=np.nan
Value to assign to the score if an error occurs in estimator fitting.
If set to 'raise', the error is raised.
@@ -1817,6 +1811,11 @@ def validation_curve(
.. versionadded:: 0.20
+ fit_params : dict, default=None
+ Parameters to pass to the fit method of the estimator.
+
+ .. versionadded:: 0.24
+
Returns
-------
train_scores : array of shape (n_ticks, n_cv_folds)
@@ -1828,7 +1827,6 @@ def validation_curve(
Notes
-----
See :ref:`sphx_glr_auto_examples_model_selection_plot_validation_curve.py`
-
"""
X, y, groups = indexable(X, y, groups)
diff --git a/sklearn/model_selection/tests/test_search.py b/sklearn/model_selection/tests/test_search.py
index 2c1a66dcb3fed..447727830d84d 100644
--- a/sklearn/model_selection/tests/test_search.py
+++ b/sklearn/model_selection/tests/test_search.py
@@ -114,7 +114,7 @@ def set_params(self, **params):
class LinearSVCNoScore(LinearSVC):
- """An LinearSVC classifier that has no score method."""
+ """A LinearSVC classifier that has no score method."""
@property
def score(self):
diff --git a/sklearn/model_selection/tests/test_validation.py b/sklearn/model_selection/tests/test_validation.py
index 90b5a605ac2e4..339f98687733d 100644
--- a/sklearn/model_selection/tests/test_validation.py
+++ b/sklearn/model_selection/tests/test_validation.py
@@ -79,7 +79,7 @@
try:
- WindowsError
+ WindowsError # type: ignore
except NameError:
WindowsError = None
diff --git a/sklearn/multioutput.py b/sklearn/multioutput.py
index 24e4cc8dda7e8..095b700e3e5f7 100644
--- a/sklearn/multioutput.py
+++ b/sklearn/multioutput.py
@@ -853,7 +853,7 @@ class RegressorChain(MetaEstimatorMixin, RegressorMixin, _BaseChain):
Parameters
----------
base_estimator : estimator
- The base estimator from which the classifier chain is built.
+ The base estimator from which the regressor chain is built.
order : array-like of shape (n_outputs,) or 'random', default=None
If `None`, the order will be determined by the order of columns in
diff --git a/sklearn/neighbors/_binary_tree.pxi b/sklearn/neighbors/_binary_tree.pxi
index 36781a770906c..ee3adfe3a8027 100644
--- a/sklearn/neighbors/_binary_tree.pxi
+++ b/sklearn/neighbors/_binary_tree.pxi
@@ -230,12 +230,14 @@ leaf_size : positive int, default=40
satisfy ``leaf_size <= n_points <= 2 * leaf_size``, except in
the case that ``n_samples < leaf_size``.
-metric : str or DistanceMetric object
- The distance metric to use for the tree. Default='minkowski'
- with p=2 (that is, a euclidean metric). See the documentation
- of the DistanceMetric class for a list of available metrics.
- {binary_tree}.valid_metrics gives a list of the metrics which
- are valid for {BinaryTree}.
+metric : str or DistanceMetric object, default='minkowski'
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2.
+ {binary_tree}.valid_metrics gives a list of the metrics which are valid for
+ {BinaryTree}. See the documentation of `scipy.spatial.distance
+ `_ and the
+ metrics listed in :class:`~sklearn.metrics.pairwise.distance_metrics` for
+ more information.
Additional keywords are passed to the distance metric class.
Note: Callable functions in the metric parameter are NOT supported for KDTree
diff --git a/sklearn/neighbors/_classification.py b/sklearn/neighbors/_classification.py
index bcad8c71aee07..254ec1af67543 100644
--- a/sklearn/neighbors/_classification.py
+++ b/sklearn/neighbors/_classification.py
@@ -9,7 +9,7 @@
# License: BSD 3 clause (C) INRIA, University of Amsterdam
import numpy as np
-from scipy import stats
+from ..utils.fixes import _mode
from ..utils.extmath import weighted_mode
from ..utils.validation import _is_arraylike, _num_samples
@@ -65,15 +65,22 @@ class KNeighborsClassifier(KNeighborsMixin, ClassifierMixin, NeighborsBase):
(l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.
metric : str or callable, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric. For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric` and the metrics listed in
- `sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS`. Note that the
- "cosine" metric uses :func:`~sklearn.metrics.pairwise.cosine_distances`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square during fit. X may be a :term:`sparse graph`,
- in which case only "nonzero" elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
+ efficient than passing the metric name as a string.
metric_params : dict, default=None
Additional keyword arguments for the metric function.
@@ -234,7 +241,7 @@ def predict(self, X):
y_pred = np.empty((n_queries, n_outputs), dtype=classes_[0].dtype)
for k, classes_k in enumerate(classes_):
if weights is None:
- mode, _ = stats.mode(_y[neigh_ind, k], axis=1)
+ mode, _ = _mode(_y[neigh_ind, k], axis=1)
else:
mode, _ = weighted_mode(_y[neigh_ind, k], weights, axis=1)
@@ -357,13 +364,22 @@ class RadiusNeighborsClassifier(RadiusNeighborsMixin, ClassifierMixin, Neighbors
(l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.
metric : str or callable, default='minkowski'
- Distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric. For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square during fit. X may be a :term:`sparse graph`,
- in which case only "nonzero" elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
+ efficient than passing the metric name as a string.
outlier_label : {manual label, 'most_frequent'}, default=None
Label for outlier samples (samples with no neighbors in given radius).
diff --git a/sklearn/neighbors/_graph.py b/sklearn/neighbors/_graph.py
index 2be70c0638517..310f3667d9fa3 100644
--- a/sklearn/neighbors/_graph.py
+++ b/sklearn/neighbors/_graph.py
@@ -65,13 +65,13 @@ def kneighbors_graph(
between neighbors according to the given metric.
metric : str, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric.
- For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric` and the metrics listed in
- `sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS`. Note that the
- "cosine" metric uses :func:`~sklearn.metrics.pairwise.cosine_distances`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
p : int, default=2
Power parameter for the Minkowski metric. When p = 1, this is
@@ -160,13 +160,13 @@ def radius_neighbors_graph(
between neighbors according to the given metric.
metric : str, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric.
- For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric` and the metrics listed in
- `sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS`. Note that the
- "cosine" metric uses :func:`~sklearn.metrics.pairwise.cosine_distances`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
p : int, default=2
Power parameter for the Minkowski metric. When p = 1, this is
@@ -266,31 +266,21 @@ class KNeighborsTransformer(
nature of the problem.
metric : str or callable, default='minkowski'
- Metric to use for distance computation. Any metric from scikit-learn
- or scipy.spatial.distance can be used.
-
- If metric is a callable function, it is called on each
- pair of instances (rows) and the resulting value recorded. The callable
- should take two arrays as input and return one value indicating the
- distance between them. This works for Scipy's metrics, but is less
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
efficient than passing the metric name as a string.
Distance matrices are not supported.
- Valid values for metric are:
-
- - from scikit-learn: ['cityblock', 'cosine', 'euclidean', 'l1', 'l2',
- 'manhattan']
-
- - from scipy.spatial.distance: ['braycurtis', 'canberra', 'chebyshev',
- 'correlation', 'dice', 'hamming', 'jaccard', 'kulsinski',
- 'mahalanobis', 'minkowski', 'rogerstanimoto', 'russellrao',
- 'seuclidean', 'sokalmichener', 'sokalsneath', 'sqeuclidean',
- 'yule']
-
- See the documentation for scipy.spatial.distance for details on these
- metrics.
-
p : int, default=2
Parameter for the Minkowski metric from
sklearn.metrics.pairwise.pairwise_distances. When p = 1, this is
@@ -493,31 +483,21 @@ class RadiusNeighborsTransformer(
nature of the problem.
metric : str or callable, default='minkowski'
- Metric to use for distance computation. Any metric from scikit-learn
- or scipy.spatial.distance can be used.
-
- If metric is a callable function, it is called on each
- pair of instances (rows) and the resulting value recorded. The callable
- should take two arrays as input and return one value indicating the
- distance between them. This works for Scipy's metrics, but is less
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
efficient than passing the metric name as a string.
Distance matrices are not supported.
- Valid values for metric are:
-
- - from scikit-learn: ['cityblock', 'cosine', 'euclidean', 'l1', 'l2',
- 'manhattan']
-
- - from scipy.spatial.distance: ['braycurtis', 'canberra', 'chebyshev',
- 'correlation', 'dice', 'hamming', 'jaccard', 'kulsinski',
- 'mahalanobis', 'minkowski', 'rogerstanimoto', 'russellrao',
- 'seuclidean', 'sokalmichener', 'sokalsneath', 'sqeuclidean',
- 'yule']
-
- See the documentation for scipy.spatial.distance for details on these
- metrics.
-
p : int, default=2
Parameter for the Minkowski metric from
sklearn.metrics.pairwise.pairwise_distances. When p = 1, this is
diff --git a/sklearn/neighbors/_kde.py b/sklearn/neighbors/_kde.py
index a785fcd86939f..d4dd498ce351d 100644
--- a/sklearn/neighbors/_kde.py
+++ b/sklearn/neighbors/_kde.py
@@ -47,12 +47,17 @@ class KernelDensity(BaseEstimator):
The kernel to use.
metric : str, default='euclidean'
- The distance metric to use. Note that not all metrics are
- valid with all algorithms. Refer to the documentation of
- :class:`BallTree` and :class:`KDTree` for a description of
- available algorithms. Note that the normalization of the density
- output is correct only for the Euclidean distance metric. Default
- is 'euclidean'.
+ Metric to use for distance computation. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
+ Not all metrics are valid with all algorithms: refer to the
+ documentation of :class:`BallTree` and :class:`KDTree`. Note that the
+ normalization of the density output is correct only for the Euclidean
+ distance metric.
atol : float, default=0
The desired absolute tolerance of the result. A larger tolerance will
diff --git a/sklearn/neighbors/_lof.py b/sklearn/neighbors/_lof.py
index 025a1c6d80768..b1d2e2a50ce27 100644
--- a/sklearn/neighbors/_lof.py
+++ b/sklearn/neighbors/_lof.py
@@ -58,34 +58,23 @@ class LocalOutlierFactor(KNeighborsMixin, OutlierMixin, NeighborsBase):
nature of the problem.
metric : str or callable, default='minkowski'
- The metric is used for distance computation. Any metric from scikit-learn
- or scipy.spatial.distance can be used.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square. X may be a sparse matrix, in which case only "nonzero"
- elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
- If metric is a callable function, it is called on each
- pair of instances (rows) and the resulting value recorded. The callable
- should take two arrays as input and return one value indicating the
- distance between them. This works for Scipy's metrics, but is less
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
efficient than passing the metric name as a string.
- Valid values for metric are:
-
- - from scikit-learn: ['cityblock', 'cosine', 'euclidean', 'l1', 'l2',
- 'manhattan']
-
- - from scipy.spatial.distance: ['braycurtis', 'canberra', 'chebyshev',
- 'correlation', 'dice', 'hamming', 'jaccard', 'kulsinski',
- 'mahalanobis', 'minkowski', 'rogerstanimoto', 'russellrao',
- 'seuclidean', 'sokalmichener', 'sokalsneath', 'sqeuclidean',
- 'yule']
-
- See the documentation for scipy.spatial.distance for details on these
- metrics:
- https://docs.scipy.org/doc/scipy/reference/spatial.distance.html.
-
p : int, default=2
Parameter for the Minkowski metric from
:func:`sklearn.metrics.pairwise.pairwise_distances`. When p = 1, this
diff --git a/sklearn/neighbors/_nca.py b/sklearn/neighbors/_nca.py
index af76a000ef2cb..96cdc3052c66e 100644
--- a/sklearn/neighbors/_nca.py
+++ b/sklearn/neighbors/_nca.py
@@ -229,7 +229,7 @@ def fit(self, X, y):
# (n_samples, n_samples)
# Initialize the transformation
- transformation = self._initialize(X, y, init)
+ transformation = np.ravel(self._initialize(X, y, init))
# Create a dictionary of parameters to be passed to the optimizer
disp = self.verbose - 2 if self.verbose > 1 else -1
diff --git a/sklearn/neighbors/_nearest_centroid.py b/sklearn/neighbors/_nearest_centroid.py
index b52d9407333a6..9403623ea8de9 100644
--- a/sklearn/neighbors/_nearest_centroid.py
+++ b/sklearn/neighbors/_nearest_centroid.py
@@ -30,11 +30,16 @@ class NearestCentroid(ClassifierMixin, BaseEstimator):
Parameters
----------
metric : str or callable, default="euclidean"
- The metric to use when calculating distance between instances in a
- feature array. If metric is a string or callable, it must be one of
- the options allowed by
- :func:`~sklearn.metrics.pairwise_distances` for its metric
- parameter. The centroids for the samples corresponding to each class is
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values. Note that "wminkowski", "seuclidean" and "mahalanobis" are not
+ supported.
+
+ The centroids for the samples corresponding to each class is
the point from which the sum of the distances (according to the metric)
of all samples that belong to that particular class are minimized.
If the `"manhattan"` metric is provided, this centroid is the median
diff --git a/sklearn/neighbors/_regression.py b/sklearn/neighbors/_regression.py
index 4c995e5062277..aef20b9baa243 100644
--- a/sklearn/neighbors/_regression.py
+++ b/sklearn/neighbors/_regression.py
@@ -72,15 +72,22 @@ class KNeighborsRegressor(KNeighborsMixin, RegressorMixin, NeighborsBase):
(l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.
metric : str or callable, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric. For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric` and the metrics listed in
- `sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS`. Note that the
- "cosine" metric uses :func:`~sklearn.metrics.pairwise.cosine_distances`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square during fit. X may be a :term:`sparse graph`,
- in which case only "nonzero" elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
+ efficient than passing the metric name as a string.
metric_params : dict, default=None
Additional keyword arguments for the metric function.
@@ -300,13 +307,22 @@ class RadiusNeighborsRegressor(RadiusNeighborsMixin, RegressorMixin, NeighborsBa
(l2) for p = 2. For arbitrary p, minkowski_distance (l_p) is used.
metric : str or callable, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric. See the documentation of :class:`DistanceMetric` for a
- list of available metrics.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square during fit. X may be a :term:`sparse graph`,
- in which case only "nonzero" elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
+ efficient than passing the metric name as a string.
metric_params : dict, default=None
Additional keyword arguments for the metric function.
diff --git a/sklearn/neighbors/_unsupervised.py b/sklearn/neighbors/_unsupervised.py
index 6399363112378..162eff0ac2a63 100644
--- a/sklearn/neighbors/_unsupervised.py
+++ b/sklearn/neighbors/_unsupervised.py
@@ -39,15 +39,22 @@ class NearestNeighbors(KNeighborsMixin, RadiusNeighborsMixin, NeighborsBase):
nature of the problem.
metric : str or callable, default='minkowski'
- The distance metric to use for the tree. The default metric is
- minkowski, and with p=2 is equivalent to the standard Euclidean
- metric. For a list of available metrics, see the documentation of
- :class:`~sklearn.metrics.DistanceMetric` and the metrics listed in
- `sklearn.metrics.pairwise.PAIRWISE_DISTANCE_FUNCTIONS`. Note that the
- "cosine" metric uses :func:`~sklearn.metrics.pairwise.cosine_distances`.
+ Metric to use for distance computation. Default is "minkowski", which
+ results in the standard Euclidean distance when p = 2. See the
+ documentation of `scipy.spatial.distance
+ `_ and
+ the metrics listed in
+ :class:`~sklearn.metrics.pairwise.distance_metrics` for valid metric
+ values.
+
If metric is "precomputed", X is assumed to be a distance matrix and
- must be square during fit. X may be a :term:`sparse graph`,
- in which case only "nonzero" elements may be considered neighbors.
+ must be square during fit. X may be a :term:`sparse graph`, in which
+ case only "nonzero" elements may be considered neighbors.
+
+ If metric is a callable function, it takes two arrays representing 1D
+ vectors as inputs and must return one value indicating the distance
+ between those vectors. This works for Scipy's metrics, but is less
+ efficient than passing the metric name as a string.
p : int, default=2
Parameter for the Minkowski metric from
diff --git a/sklearn/neighbors/tests/test_neighbors.py b/sklearn/neighbors/tests/test_neighbors.py
index 6b303886e2b5a..6ffd49422b1a3 100644
--- a/sklearn/neighbors/tests/test_neighbors.py
+++ b/sklearn/neighbors/tests/test_neighbors.py
@@ -70,7 +70,7 @@
ALGORITHMS = ("ball_tree", "brute", "kd_tree", "auto")
COMMON_VALID_METRICS = sorted(
set.intersection(*map(set, neighbors.VALID_METRICS.values()))
-)
+) # type: ignore
P = (1, 2, 3, 4, np.inf)
JOBLIB_BACKENDS = list(joblib.parallel.BACKENDS.keys())
diff --git a/sklearn/pipeline.py b/sklearn/pipeline.py
index 74347f250bc83..086bc4e1f0557 100644
--- a/sklearn/pipeline.py
+++ b/sklearn/pipeline.py
@@ -75,8 +75,8 @@ class Pipeline(_BaseComposition):
----------
steps : list of tuple
List of (name, transform) tuples (implementing `fit`/`transform`) that
- are chained, in the order in which they are chained, with the last
- object an estimator.
+ are chained in sequential order. The last transform must be an
+ estimator.
memory : str or object with the joblib.Memory interface, default=None
Used to cache the fitted transformers of the pipeline. By default,
@@ -1244,8 +1244,7 @@ def _sk_visual_block_(self):
def make_union(*transformers, n_jobs=None, verbose=False):
- """
- Construct a FeatureUnion from the given transformers.
+ """Construct a FeatureUnion from the given transformers.
This is a shorthand for the FeatureUnion constructor; it does not require,
and does not permit, naming the transformers. Instead, they will be given
@@ -1254,6 +1253,7 @@ def make_union(*transformers, n_jobs=None, verbose=False):
Parameters
----------
*transformers : list of estimators
+ One or more estimators.
n_jobs : int, default=None
Number of jobs to run in parallel.
@@ -1262,7 +1262,7 @@ def make_union(*transformers, n_jobs=None, verbose=False):
for more details.
.. versionchanged:: v0.20
- `n_jobs` default changed from 1 to None
+ `n_jobs` default changed from 1 to None.
verbose : bool, default=False
If True, the time elapsed while fitting each transformer will be
@@ -1271,6 +1271,8 @@ def make_union(*transformers, n_jobs=None, verbose=False):
Returns
-------
f : FeatureUnion
+ A :class:`FeatureUnion` object for concatenating the results of multiple
+ transformer objects.
See Also
--------
diff --git a/sklearn/preprocessing/_data.py b/sklearn/preprocessing/_data.py
index f0088aab521ad..410501dfbcebd 100644
--- a/sklearn/preprocessing/_data.py
+++ b/sklearn/preprocessing/_data.py
@@ -1659,6 +1659,11 @@ def robust_scale(
X_tr : {ndarray, sparse matrix} of shape (n_samples, n_features)
The transformed data.
+ See Also
+ --------
+ RobustScaler : Performs centering and scaling using the Transformer API
+ (e.g. as part of a preprocessing :class:`~sklearn.pipeline.Pipeline`).
+
Notes
-----
This implementation will refuse to center scipy.sparse matrices
@@ -1687,11 +1692,6 @@ def robust_scale(
:class:`~sklearn.preprocessing.RobustScaler` within a
:ref:`Pipeline ` in order to prevent most risks of data
leaking: `pipe = make_pipeline(RobustScaler(), LogisticRegression())`.
-
- See Also
- --------
- RobustScaler : Performs centering and scaling using the Transformer API
- (e.g. as part of a preprocessing :class:`~sklearn.pipeline.Pipeline`).
"""
X = check_array(
X,
@@ -3293,8 +3293,8 @@ def _check_input(
reset=in_fit,
)
- with np.warnings.catch_warnings():
- np.warnings.filterwarnings("ignore", r"All-NaN (slice|axis) encountered")
+ with warnings.catch_warnings():
+ warnings.filterwarnings("ignore", r"All-NaN (slice|axis) encountered")
if check_positive and self.method == "box-cox" and np.nanmin(X) <= 0:
raise ValueError(
"The Box-Cox transformation can only be "
diff --git a/sklearn/preprocessing/_encoders.py b/sklearn/preprocessing/_encoders.py
index d4cc642a18562..0aad790d8b098 100644
--- a/sklearn/preprocessing/_encoders.py
+++ b/sklearn/preprocessing/_encoders.py
@@ -664,7 +664,7 @@ def _fit_infrequent_category_mapping(self, n_samples, category_counts):
to a single output:
`_default_to_infrequent_mappings[7] = array([0, 3, 1, 3, 2, 3])`
- Defines private attrite: `_infrequent_indices`. `_infrequent_indices[i]`
+ Defines private attribute: `_infrequent_indices`. `_infrequent_indices[i]`
is an array of indices such that
`categories_[i][_infrequent_indices[i]]` are all the infrequent category
labels. If the feature `i` has no infrequent categories
@@ -1408,19 +1408,23 @@ def inverse_transform(self, X):
found_unknown = {}
for i in range(n_features):
- labels = X[:, i].astype("int64", copy=False)
+ labels = X[:, i]
# replace values of X[:, i] that were nan with actual indices
if i in self._missing_indices:
- X_i_mask = _get_mask(X[:, i], self.encoded_missing_value)
+ X_i_mask = _get_mask(labels, self.encoded_missing_value)
labels[X_i_mask] = self._missing_indices[i]
if self.handle_unknown == "use_encoded_value":
- unknown_labels = labels == self.unknown_value
- X_tr[:, i] = self.categories_[i][np.where(unknown_labels, 0, labels)]
+ unknown_labels = _get_mask(labels, self.unknown_value)
+
+ known_labels = ~unknown_labels
+ X_tr[known_labels, i] = self.categories_[i][
+ labels[known_labels].astype("int64", copy=False)
+ ]
found_unknown[i] = unknown_labels
else:
- X_tr[:, i] = self.categories_[i][labels]
+ X_tr[:, i] = self.categories_[i][labels.astype("int64", copy=False)]
# insert None values for unknown values
if found_unknown:
diff --git a/sklearn/preprocessing/_label.py b/sklearn/preprocessing/_label.py
index e7f4a5e337208..caf887e2bb648 100644
--- a/sklearn/preprocessing/_label.py
+++ b/sklearn/preprocessing/_label.py
@@ -447,6 +447,11 @@ def label_binarize(y, *, classes, neg_label=0, pos_label=1, sparse_output=False)
Shape will be (n_samples, 1) for binary problems. Sparse matrix will
be of CSR format.
+ See Also
+ --------
+ LabelBinarizer : Class used to wrap the functionality of label_binarize and
+ allow for fitting to classes independently of the transform operation.
+
Examples
--------
>>> from sklearn.preprocessing import label_binarize
@@ -467,11 +472,6 @@ def label_binarize(y, *, classes, neg_label=0, pos_label=1, sparse_output=False)
[0],
[0],
[1]])
-
- See Also
- --------
- LabelBinarizer : Class used to wrap the functionality of label_binarize and
- allow for fitting to classes independently of the transform operation.
"""
if not isinstance(y, list):
# XXX Workaround that will be removed when list of list format is
diff --git a/sklearn/preprocessing/tests/test_encoders.py b/sklearn/preprocessing/tests/test_encoders.py
index ea32de22cd2f0..2a90f9894f073 100644
--- a/sklearn/preprocessing/tests/test_encoders.py
+++ b/sklearn/preprocessing/tests/test_encoders.py
@@ -1928,6 +1928,15 @@ def test_ordinal_encoder_unknown_missing_interaction():
X_test_trans = oe.transform(X_test)
assert_allclose(X_test_trans, [[np.nan], [-3]])
+ # Non-regression test for #24082
+ X_roundtrip = oe.inverse_transform(X_test_trans)
+
+ # np.nan is unknown so it maps to None
+ assert X_roundtrip[0][0] is None
+
+ # -3 is the encoded missing value so it maps back to nan
+ assert np.isnan(X_roundtrip[1][0])
+
@pytest.mark.parametrize("with_pandas", [True, False])
def test_ordinal_encoder_encoded_missing_value_error(with_pandas):
@@ -1953,3 +1962,55 @@ def test_ordinal_encoder_encoded_missing_value_error(with_pandas):
with pytest.raises(ValueError, match=error_msg):
oe.fit(X)
+
+
+@pytest.mark.parametrize(
+ "X_train, X_test_trans_expected, X_roundtrip_expected",
+ [
+ (
+ # missing value is not in training set
+ # inverse transform will considering encoded nan as unknown
+ np.array([["a"], ["1"]], dtype=object),
+ [[0], [np.nan], [np.nan]],
+ np.asarray([["1"], [None], [None]], dtype=object),
+ ),
+ (
+ # missing value in training set,
+ # inverse transform will considering encoded nan as missing
+ np.array([[np.nan], ["1"], ["a"]], dtype=object),
+ [[0], [np.nan], [np.nan]],
+ np.asarray([["1"], [np.nan], [np.nan]], dtype=object),
+ ),
+ ],
+)
+def test_ordinal_encoder_unknown_missing_interaction_both_nan(
+ X_train, X_test_trans_expected, X_roundtrip_expected
+):
+ """Check transform when unknown_value and encoded_missing_value is nan.
+
+ Non-regression test for #24082.
+ """
+ oe = OrdinalEncoder(
+ handle_unknown="use_encoded_value",
+ unknown_value=np.nan,
+ encoded_missing_value=np.nan,
+ ).fit(X_train)
+
+ X_test = np.array([["1"], [np.nan], ["b"]])
+ X_test_trans = oe.transform(X_test)
+
+ # both nan and unknown are encoded as nan
+ assert_allclose(X_test_trans, X_test_trans_expected)
+ X_roundtrip = oe.inverse_transform(X_test_trans)
+
+ n_samples = X_roundtrip_expected.shape[0]
+ for i in range(n_samples):
+ expected_val = X_roundtrip_expected[i, 0]
+ val = X_roundtrip[i, 0]
+
+ if expected_val is None:
+ assert val is None
+ elif is_scalar_nan(expected_val):
+ assert np.isnan(val)
+ else:
+ assert val == expected_val
diff --git a/sklearn/random_projection.py b/sklearn/random_projection.py
index 3b4a5e2236db5..7bda89bfec3b2 100644
--- a/sklearn/random_projection.py
+++ b/sklearn/random_projection.py
@@ -94,6 +94,15 @@ def johnson_lindenstrauss_min_dim(n_samples, *, eps=0.1):
The minimal number of components to guarantee with good probability
an eps-embedding with n_samples.
+ References
+ ----------
+
+ .. [1] https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma
+
+ .. [2] Sanjoy Dasgupta and Anupam Gupta, 1999,
+ "An elementary proof of the Johnson-Lindenstrauss Lemma."
+ http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.3654
+
Examples
--------
>>> from sklearn.random_projection import johnson_lindenstrauss_min_dim
@@ -105,16 +114,6 @@ def johnson_lindenstrauss_min_dim(n_samples, *, eps=0.1):
>>> johnson_lindenstrauss_min_dim([1e4, 1e5, 1e6], eps=0.1)
array([ 7894, 9868, 11841])
-
- References
- ----------
-
- .. [1] https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma
-
- .. [2] Sanjoy Dasgupta and Anupam Gupta, 1999,
- "An elementary proof of the Johnson-Lindenstrauss Lemma."
- http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.3654
-
"""
eps = np.asarray(eps)
n_samples = np.asarray(n_samples)
@@ -250,7 +249,7 @@ def _sparse_random_matrix(n_components, n_features, density="auto", random_state
https://web.stanford.edu/~hastie/Papers/Ping/KDD06_rp.pdf
.. [2] D. Achlioptas, 2001, "Database-friendly random projections",
- http://www.cs.ucsc.edu/~optas/papers/jl.pdf
+ https://cgi.di.uoa.gr/~optas/papers/jl.pdf
"""
_check_input_size(n_components, n_features)
@@ -710,7 +709,7 @@ class SparseRandomProjection(BaseRandomProjection):
https://web.stanford.edu/~hastie/Papers/Ping/KDD06_rp.pdf
.. [2] D. Achlioptas, 2001, "Database-friendly random projections",
- https://users.soe.ucsc.edu/~optas/papers/jl.pdf
+ https://cgi.di.uoa.gr/~optas/papers/jl.pdf
Examples
--------
diff --git a/sklearn/svm/_classes.py b/sklearn/svm/_classes.py
index 3cfbafce876ea..42e696875728d 100644
--- a/sklearn/svm/_classes.py
+++ b/sklearn/svm/_classes.py
@@ -1167,8 +1167,8 @@ class SVR(RegressorMixin, BaseLibSVM):
.. versionadded:: 1.1
- n_support_ : ndarray of shape (n_classes,), dtype=int32
- Number of support vectors for each class.
+ n_support_ : ndarray of shape (1,), dtype=int32
+ Number of support vectors.
shape_fit_ : tuple of int of shape (n_dimensions_of_X,)
Array dimensions of training vector ``X``.
@@ -1359,8 +1359,8 @@ class NuSVR(RegressorMixin, BaseLibSVM):
.. versionadded:: 1.1
- n_support_ : ndarray of shape (n_classes,), dtype=int32
- Number of support vectors for each class.
+ n_support_ : ndarray of shape (1,), dtype=int32
+ Number of support vectors.
shape_fit_ : tuple of int of shape (n_dimensions_of_X,)
Array dimensions of training vector ``X``.
diff --git a/sklearn/svm/tests/test_svm.py b/sklearn/svm/tests/test_svm.py
index db1d49ab4bcf9..dccd18217e710 100644
--- a/sklearn/svm/tests/test_svm.py
+++ b/sklearn/svm/tests/test_svm.py
@@ -4,7 +4,6 @@
TODO: remove hard coded numerical results when possible
"""
import numpy as np
-import itertools
import pytest
import re
@@ -828,36 +827,29 @@ def test_sparse_fit_support_vectors_empty():
assert not model.dual_coef_.data.size
-def test_linearsvc_parameters():
+@pytest.mark.parametrize("loss", ["hinge", "squared_hinge"])
+@pytest.mark.parametrize("penalty", ["l1", "l2"])
+@pytest.mark.parametrize("dual", [True, False])
+def test_linearsvc_parameters(loss, penalty, dual):
# Test possible parameter combinations in LinearSVC
# Generate list of possible parameter combinations
- losses = ["hinge", "squared_hinge", "logistic_regression", "foo"]
- penalties, duals = ["l1", "l2", "bar"], [True, False]
-
- X, y = make_classification(n_samples=5, n_features=5)
-
- for loss, penalty, dual in itertools.product(losses, penalties, duals):
- clf = svm.LinearSVC(penalty=penalty, loss=loss, dual=dual)
- if (
- (loss, penalty) == ("hinge", "l1")
- or (loss, penalty, dual) == ("hinge", "l2", False)
- or (penalty, dual) == ("l1", True)
- or loss == "foo"
- or penalty == "bar"
- ):
+ X, y = make_classification(n_samples=5, n_features=5, random_state=0)
- with pytest.raises(
- ValueError,
- match="Unsupported set of arguments.*penalty='%s.*loss='%s.*dual=%s"
- % (penalty, loss, dual),
- ):
- clf.fit(X, y)
- else:
- clf.fit(X, y)
+ clf = svm.LinearSVC(penalty=penalty, loss=loss, dual=dual, random_state=0)
+ if (
+ (loss, penalty) == ("hinge", "l1")
+ or (loss, penalty, dual) == ("hinge", "l2", False)
+ or (penalty, dual) == ("l1", True)
+ ):
- # Incorrect loss value - test if explicit error message is raised
- with pytest.raises(ValueError, match=".*loss='l3' is not supported.*"):
- svm.LinearSVC(loss="l3").fit(X, y)
+ with pytest.raises(
+ ValueError,
+ match="Unsupported set of arguments.*penalty='%s.*loss='%s.*dual=%s"
+ % (penalty, loss, dual),
+ ):
+ clf.fit(X, y)
+ else:
+ clf.fit(X, y)
def test_linear_svx_uppercase_loss_penality_raises_error():
@@ -1398,23 +1390,18 @@ def test_linearsvm_liblinear_sample_weight(SVM, params):
assert_allclose(X_est_no_weight, X_est_with_weight)
-def test_n_support_oneclass_svr():
+@pytest.mark.parametrize("Klass", (svm.OneClassSVM, svm.SVR, svm.NuSVR))
+def test_n_support(Klass):
# Make n_support is correct for oneclass and SVR (used to be
# non-initialized)
# this is a non regression test for issue #14774
X = np.array([[0], [0.44], [0.45], [0.46], [1]])
- clf = svm.OneClassSVM()
- assert not hasattr(clf, "n_support_")
- clf.fit(X)
- assert clf.n_support_ == clf.support_vectors_.shape[0]
- assert clf.n_support_.size == 1
- assert clf.n_support_ == 3
-
y = np.arange(X.shape[0])
- reg = svm.SVR().fit(X, y)
- assert reg.n_support_ == reg.support_vectors_.shape[0]
- assert reg.n_support_.size == 1
- assert reg.n_support_ == 4
+ est = Klass()
+ assert not hasattr(est, "n_support_")
+ est.fit(X, y)
+ assert est.n_support_[0] == est.support_vectors_.shape[0]
+ assert est.n_support_.size == 1
@pytest.mark.parametrize("Estimator", [svm.SVC, svm.SVR])
diff --git a/sklearn/tests/test_base.py b/sklearn/tests/test_base.py
index bdbe55c463841..3e1915c10af79 100644
--- a/sklearn/tests/test_base.py
+++ b/sklearn/tests/test_base.py
@@ -229,7 +229,7 @@ def test_str():
def test_get_params():
- test = T(K(), K())
+ test = T(K(), K)
assert "a__d" in test.get_params(deep=True)
assert "a__d" not in test.get_params(deep=False)
@@ -653,9 +653,9 @@ def transform(self, X):
"Feature names only support names that are all strings. "
"Got feature names with dtypes: ['int', 'str']"
)
- with pytest.warns(FutureWarning, match=msg) as record:
+ with pytest.warns(FutureWarning, match=msg):
trans.fit(df_mixed)
# transform on feature names that are mixed also warns:
- with pytest.warns(FutureWarning, match=msg) as record:
+ with pytest.warns(FutureWarning, match=msg):
trans.transform(df_mixed)
diff --git a/sklearn/tests/test_common.py b/sklearn/tests/test_common.py
index b5fc83d1028b3..922573f7e3fc8 100644
--- a/sklearn/tests/test_common.py
+++ b/sklearn/tests/test_common.py
@@ -18,6 +18,24 @@
import pytest
import numpy as np
+from sklearn.cluster import (
+ AffinityPropagation,
+ Birch,
+ MeanShift,
+ OPTICS,
+ SpectralClustering,
+)
+from sklearn.datasets import make_blobs
+from sklearn.manifold import Isomap, TSNE, LocallyLinearEmbedding
+from sklearn.neighbors import (
+ LocalOutlierFactor,
+ KNeighborsClassifier,
+ KNeighborsRegressor,
+ RadiusNeighborsClassifier,
+ RadiusNeighborsRegressor,
+)
+from sklearn.semi_supervised import LabelPropagation, LabelSpreading
+
from sklearn.utils import all_estimators
from sklearn.utils._testing import ignore_warnings
from sklearn.exceptions import ConvergenceWarning
@@ -442,3 +460,44 @@ def test_estimators_do_not_raise_errors_in_init_or_set_params(Estimator):
# Also do does not raise
est.set_params(**new_params)
+
+
+# TODO: remove this filter in 1.2
+@pytest.mark.filterwarnings("ignore::FutureWarning:sklearn")
+@pytest.mark.parametrize(
+ "Estimator",
+ [
+ AffinityPropagation,
+ Birch,
+ MeanShift,
+ KNeighborsClassifier,
+ KNeighborsRegressor,
+ RadiusNeighborsClassifier,
+ RadiusNeighborsRegressor,
+ LabelPropagation,
+ LabelSpreading,
+ OPTICS,
+ SpectralClustering,
+ LocalOutlierFactor,
+ LocallyLinearEmbedding,
+ Isomap,
+ TSNE,
+ ],
+)
+def test_f_contiguous_array_estimator(Estimator):
+ # Non-regression test for:
+ # https://github.com/scikit-learn/scikit-learn/issues/23988
+ # https://github.com/scikit-learn/scikit-learn/issues/24013
+
+ X, _ = make_blobs(n_samples=80, n_features=4, random_state=0)
+ X = np.asfortranarray(X)
+ y = np.round(X[:, 0])
+
+ est = Estimator()
+ est.fit(X, y)
+
+ if hasattr(est, "transform"):
+ est.transform(X)
+
+ if hasattr(est, "predict"):
+ est.predict(X)
diff --git a/sklearn/tests/test_docstring_parameters.py b/sklearn/tests/test_docstring_parameters.py
index 5e22b425be1ec..4503446823b2a 100644
--- a/sklearn/tests/test_docstring_parameters.py
+++ b/sklearn/tests/test_docstring_parameters.py
@@ -248,7 +248,7 @@ def test_fit_docstring_attributes(name, Estimator):
# FIXME: TO BE REMOVED for 1.2 (avoid FutureWarning)
if Estimator.__name__ == "TSNE":
- est.set_params(learning_rate=200.0, init="random")
+ est.set_params(learning_rate=200.0, init="random", perplexity=2)
# FIXME: TO BE REMOVED for 1.3 (avoid FutureWarning)
if Estimator.__name__ == "SequentialFeatureSelector":
diff --git a/sklearn/tests/test_docstrings.py b/sklearn/tests/test_docstrings.py
index 0131dae6c01a3..88d37ef7b232d 100644
--- a/sklearn/tests/test_docstrings.py
+++ b/sklearn/tests/test_docstrings.py
@@ -12,11 +12,7 @@
numpydoc_validation = pytest.importorskip("numpydoc.validate")
FUNCTION_DOCSTRING_IGNORE_LIST = [
- "sklearn.datasets._kddcup99.fetch_kddcup99",
- "sklearn.datasets._lfw.fetch_lfw_pairs",
"sklearn.datasets._lfw.fetch_lfw_people",
- "sklearn.datasets._samples_generator.make_gaussian_quantiles",
- "sklearn.datasets._samples_generator.make_spd_matrix",
"sklearn.datasets._species_distributions.fetch_species_distributions",
"sklearn.datasets._svmlight_format_io.load_svmlight_file",
"sklearn.datasets._svmlight_format_io.load_svmlight_files",
@@ -24,28 +20,17 @@
"sklearn.decomposition._dict_learning.dict_learning_online",
"sklearn.decomposition._nmf.non_negative_factorization",
"sklearn.externals._packaging.version.parse",
- "sklearn.feature_extraction.image.extract_patches_2d",
"sklearn.feature_extraction.text.strip_accents_unicode",
- "sklearn.feature_selection._univariate_selection.chi2",
- "sklearn.feature_selection._univariate_selection.f_oneway",
"sklearn.inspection._partial_dependence.partial_dependence",
"sklearn.inspection._plot.partial_dependence.plot_partial_dependence",
"sklearn.linear_model._least_angle.lars_path_gram",
"sklearn.linear_model._omp.orthogonal_mp_gram",
"sklearn.manifold._locally_linear.locally_linear_embedding",
"sklearn.manifold._t_sne.trustworthiness",
- "sklearn.metrics._classification.brier_score_loss",
- "sklearn.metrics._classification.cohen_kappa_score",
- "sklearn.metrics._classification.fbeta_score",
- "sklearn.metrics._classification.jaccard_score",
- "sklearn.metrics._classification.log_loss",
"sklearn.metrics._plot.det_curve.plot_det_curve",
"sklearn.metrics._plot.precision_recall_curve.plot_precision_recall_curve",
- "sklearn.metrics._ranking.auc",
"sklearn.metrics._ranking.coverage_error",
"sklearn.metrics._ranking.dcg_score",
- "sklearn.metrics._ranking.label_ranking_average_precision_score",
- "sklearn.metrics._ranking.roc_auc_score",
"sklearn.metrics._ranking.roc_curve",
"sklearn.metrics._ranking.top_k_accuracy_score",
"sklearn.metrics._regression.mean_pinball_loss",
@@ -54,37 +39,11 @@
"sklearn.metrics.cluster._supervised.adjusted_rand_score",
"sklearn.metrics.cluster._supervised.entropy",
"sklearn.metrics.cluster._supervised.fowlkes_mallows_score",
- "sklearn.metrics.cluster._supervised.homogeneity_completeness_v_measure",
- "sklearn.metrics.cluster._supervised.mutual_info_score",
- "sklearn.metrics.cluster._supervised.normalized_mutual_info_score",
- "sklearn.metrics.cluster._supervised.pair_confusion_matrix",
"sklearn.metrics.cluster._supervised.rand_score",
"sklearn.metrics.cluster._supervised.v_measure_score",
- "sklearn.metrics.pairwise.additive_chi2_kernel",
- "sklearn.metrics.pairwise.check_paired_arrays",
- "sklearn.metrics.pairwise.check_pairwise_arrays",
- "sklearn.metrics.pairwise.chi2_kernel",
- "sklearn.metrics.pairwise.cosine_distances",
- "sklearn.metrics.pairwise.cosine_similarity",
- "sklearn.metrics.pairwise.distance_metrics",
- "sklearn.metrics.pairwise.kernel_metrics",
- "sklearn.metrics.pairwise.paired_manhattan_distances",
- "sklearn.metrics.pairwise.pairwise_distances_argmin",
- "sklearn.metrics.pairwise.pairwise_distances_argmin_min",
"sklearn.metrics.pairwise.pairwise_distances_chunked",
- "sklearn.metrics.pairwise.pairwise_kernels",
- "sklearn.metrics.pairwise.polynomial_kernel",
- "sklearn.metrics.pairwise.rbf_kernel",
- "sklearn.metrics.pairwise.sigmoid_kernel",
- "sklearn.model_selection._validation.learning_curve",
- "sklearn.model_selection._validation.permutation_test_score",
- "sklearn.model_selection._validation.validation_curve",
- "sklearn.pipeline.make_union",
"sklearn.preprocessing._data.maxabs_scale",
- "sklearn.preprocessing._data.robust_scale",
"sklearn.preprocessing._data.scale",
- "sklearn.preprocessing._label.label_binarize",
- "sklearn.random_projection.johnson_lindenstrauss_min_dim",
"sklearn.svm._bounds.l1_min_c",
"sklearn.tree._export.plot_tree",
"sklearn.utils.axis0_safe_slice",
@@ -112,14 +71,12 @@
"sklearn.utils.multiclass.class_distribution",
"sklearn.utils.multiclass.type_of_target",
"sklearn.utils.multiclass.unique_labels",
- "sklearn.utils.resample",
"sklearn.utils.safe_mask",
"sklearn.utils.safe_sqr",
"sklearn.utils.shuffle",
"sklearn.utils.sparsefuncs.count_nonzero",
"sklearn.utils.sparsefuncs.csc_median_axis_0",
"sklearn.utils.sparsefuncs.incr_mean_variance_axis",
- "sklearn.utils.sparsefuncs.inplace_swap_column",
"sklearn.utils.sparsefuncs.inplace_swap_row",
"sklearn.utils.sparsefuncs.inplace_swap_row_csc",
"sklearn.utils.sparsefuncs.inplace_swap_row_csr",
@@ -144,7 +101,7 @@ def get_all_methods():
methods.append(name)
methods.append(None)
- for method in sorted(methods, key=lambda x: str(x)):
+ for method in sorted(methods, key=str):
yield Estimator, method
diff --git a/sklearn/tests/test_isotonic.py b/sklearn/tests/test_isotonic.py
index 5600cf8706e75..0b646439a3b41 100644
--- a/sklearn/tests/test_isotonic.py
+++ b/sklearn/tests/test_isotonic.py
@@ -336,7 +336,7 @@ def test_isotonic_regression_oob_raise():
ir.fit(x, y)
# Check that an exception is thrown
- msg = "A value in x_new is below the interpolation range"
+ msg = "in x_new is below the interpolation range"
with pytest.raises(ValueError, match=msg):
ir.predict([min(x) - 10, max(x) + 10])
diff --git a/sklearn/tree/_classes.py b/sklearn/tree/_classes.py
index 79257355a4150..c6e528c4f50f6 100644
--- a/sklearn/tree/_classes.py
+++ b/sklearn/tree/_classes.py
@@ -729,8 +729,8 @@ class DecisionTreeClassifier(ClassifierMixin, BaseDecisionTree):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `int(max_features * n_features)` features are considered at each
- split.
+ `max(1, int(max_features * n_features_in_))` features are considered at
+ each split.
- If "auto", then `max_features=sqrt(n_features)`.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
@@ -1140,7 +1140,7 @@ class DecisionTreeRegressor(RegressorMixin, BaseDecisionTree):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `int(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=n_features`.
- If "sqrt", then `max_features=sqrt(n_features)`.
@@ -1450,8 +1450,8 @@ class ExtraTreeClassifier(DecisionTreeClassifier):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `int(max_features * n_features)` features are considered at each
- split.
+ `max(1, int(max_features * n_features_in_))` features are considered at
+ each split.
- If "auto", then `max_features=sqrt(n_features)`.
- If "sqrt", then `max_features=sqrt(n_features)`.
- If "log2", then `max_features=log2(n_features)`.
@@ -1730,7 +1730,7 @@ class ExtraTreeRegressor(DecisionTreeRegressor):
- If int, then consider `max_features` features at each split.
- If float, then `max_features` is a fraction and
- `int(max_features * n_features)` features are considered at each
+ `max(1, int(max_features * n_features_in_))` features are considered at each
split.
- If "auto", then `max_features=n_features`.
- If "sqrt", then `max_features=sqrt(n_features)`.
diff --git a/sklearn/tree/_criterion.pxd b/sklearn/tree/_criterion.pxd
index 1639b5f4b3195..bc78c09b6ff5e 100644
--- a/sklearn/tree/_criterion.pxd
+++ b/sklearn/tree/_criterion.pxd
@@ -9,9 +9,6 @@
# See _criterion.pyx for implementation details.
-import numpy as np
-cimport numpy as np
-
from ._tree cimport DTYPE_t # Type of X
from ._tree cimport DOUBLE_t # Type of y, sample_weight
from ._tree cimport SIZE_t # Type for indices and counters
diff --git a/sklearn/tree/_criterion.pyx b/sklearn/tree/_criterion.pyx
index 57012fcab2296..680719464e11d 100644
--- a/sklearn/tree/_criterion.pyx
+++ b/sklearn/tree/_criterion.pyx
@@ -17,8 +17,8 @@ from libc.string cimport memset
from libc.math cimport fabs
import numpy as np
-cimport numpy as np
-np.import_array()
+cimport numpy as cnp
+cnp.import_array()
from numpy.math cimport INFINITY
from scipy.special.cython_special cimport xlogy
@@ -197,7 +197,7 @@ cdef class ClassificationCriterion(Criterion):
"""Abstract criterion for classification."""
def __cinit__(self, SIZE_t n_outputs,
- np.ndarray[SIZE_t, ndim=1] n_classes):
+ cnp.ndarray[SIZE_t, ndim=1] n_classes):
"""Initialize attributes for this criterion.
Parameters
@@ -874,8 +874,8 @@ cdef class MAE(RegressionCriterion):
MAE = (1 / n)*(\sum_i |y_i - f_i|), where y_i is the true
value and f_i is the predicted value."""
- cdef np.ndarray left_child
- cdef np.ndarray right_child
+ cdef cnp.ndarray left_child
+ cdef cnp.ndarray right_child
cdef DOUBLE_t[::1] node_medians
def __cinit__(self, SIZE_t n_outputs, SIZE_t n_samples):
diff --git a/sklearn/tree/_splitter.pxd b/sklearn/tree/_splitter.pxd
index cf01fed9cfd7d..1b899d2c66454 100644
--- a/sklearn/tree/_splitter.pxd
+++ b/sklearn/tree/_splitter.pxd
@@ -9,9 +9,6 @@
# See _splitter.pyx for details.
-import numpy as np
-cimport numpy as np
-
from ._criterion cimport Criterion
from ._tree cimport DTYPE_t # Type of X
@@ -46,13 +43,13 @@ cdef class Splitter:
cdef object random_state # Random state
cdef UINT32_t rand_r_state # sklearn_rand_r random number state
- cdef SIZE_t* samples # Sample indices in X, y
+ cdef SIZE_t[::1] samples # Sample indices in X, y
cdef SIZE_t n_samples # X.shape[0]
cdef double weighted_n_samples # Weighted number of samples
- cdef SIZE_t* features # Feature indices in X
- cdef SIZE_t* constant_features # Constant features indices
+ cdef SIZE_t[::1] features # Feature indices in X
+ cdef SIZE_t[::1] constant_features # Constant features indices
cdef SIZE_t n_features # X.shape[1]
- cdef DTYPE_t* feature_values # temp. array holding feature values
+ cdef DTYPE_t[::1] feature_values # temp. array holding feature values
cdef SIZE_t start # Start position for the current node
cdef SIZE_t end # End position for the current node
diff --git a/sklearn/tree/_splitter.pyx b/sklearn/tree/_splitter.pyx
index 514975380a8b5..76b502f98f144 100644
--- a/sklearn/tree/_splitter.pyx
+++ b/sklearn/tree/_splitter.pyx
@@ -19,8 +19,6 @@ from libc.string cimport memcpy
from libc.string cimport memset
import numpy as np
-cimport numpy as np
-np.import_array()
from scipy.sparse import csc_matrix
@@ -28,7 +26,6 @@ from ._utils cimport log
from ._utils cimport rand_int
from ._utils cimport rand_uniform
from ._utils cimport RAND_R_MAX
-from ._utils cimport safe_realloc
cdef double INFINITY = np.inf
@@ -82,11 +79,8 @@ cdef class Splitter:
self.criterion = criterion
- self.samples = NULL
self.n_samples = 0
- self.features = NULL
self.n_features = 0
- self.feature_values = NULL
self.sample_weight = NULL
@@ -95,14 +89,6 @@ cdef class Splitter:
self.min_weight_leaf = min_weight_leaf
self.random_state = random_state
- def __dealloc__(self):
- """Destructor."""
-
- free(self.samples)
- free(self.features)
- free(self.constant_features)
- free(self.feature_values)
-
def __getstate__(self):
return {}
@@ -139,7 +125,8 @@ cdef class Splitter:
# Create a new array which will be used to store nonzero
# samples from the feature of interest
- cdef SIZE_t* samples = safe_realloc(&self.samples, n_samples)
+ self.samples = np.empty(n_samples, dtype=np.intp)
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t i, j
cdef double weighted_n_samples = 0.0
@@ -161,15 +148,11 @@ cdef class Splitter:
self.weighted_n_samples = weighted_n_samples
cdef SIZE_t n_features = X.shape[1]
- cdef SIZE_t* features = safe_realloc(&self.features, n_features)
-
- for i in range(n_features):
- features[i] = i
-
+ self.features = np.arange(n_features, dtype=np.intp)
self.n_features = n_features
- safe_realloc(&self.feature_values, n_samples)
- safe_realloc(&self.constant_features, n_features)
+ self.feature_values = np.empty(n_samples, dtype=np.float32)
+ self.constant_features = np.empty(n_features, dtype=np.intp)
self.y = y
@@ -199,7 +182,7 @@ cdef class Splitter:
self.criterion.init(self.y,
self.sample_weight,
self.weighted_n_samples,
- self.samples,
+ &self.samples[0],
start,
end)
@@ -268,15 +251,15 @@ cdef class BestSplitter(BaseDenseSplitter):
or 0 otherwise.
"""
# Find the best split
- cdef SIZE_t* samples = self.samples
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t start = self.start
cdef SIZE_t end = self.end
- cdef SIZE_t* features = self.features
- cdef SIZE_t* constant_features = self.constant_features
+ cdef SIZE_t[::1] features = self.features
+ cdef SIZE_t[::1] constant_features = self.constant_features
cdef SIZE_t n_features = self.n_features
- cdef DTYPE_t* Xf = self.feature_values
+ cdef DTYPE_t[::1] Xf = self.feature_values
cdef SIZE_t max_features = self.max_features
cdef SIZE_t min_samples_leaf = self.min_samples_leaf
cdef double min_weight_leaf = self.min_weight_leaf
@@ -344,75 +327,78 @@ cdef class BestSplitter(BaseDenseSplitter):
features[n_drawn_constants], features[f_j] = features[f_j], features[n_drawn_constants]
n_drawn_constants += 1
+ continue
- else:
- # f_j in the interval [n_known_constants, f_i - n_found_constants[
- f_j += n_found_constants
- # f_j in the interval [n_total_constants, f_i[
- current.feature = features[f_j]
+ # f_j in the interval [n_known_constants, f_i - n_found_constants[
+ f_j += n_found_constants
+ # f_j in the interval [n_total_constants, f_i[
+ current.feature = features[f_j]
- # Sort samples along that feature; by
- # copying the values into an array and
- # sorting the array in a manner which utilizes the cache more
- # effectively.
- for i in range(start, end):
- Xf[i] = self.X[samples[i], current.feature]
+ # Sort samples along that feature; by
+ # copying the values into an array and
+ # sorting the array in a manner which utilizes the cache more
+ # effectively.
+ for i in range(start, end):
+ Xf[i] = self.X[samples[i], current.feature]
- sort(Xf + start, samples + start, end - start)
+ sort(&Xf[start], &samples[start], end - start)
- if Xf[end - 1] <= Xf[start] + FEATURE_THRESHOLD:
- features[f_j], features[n_total_constants] = features[n_total_constants], features[f_j]
+ if Xf[end - 1] <= Xf[start] + FEATURE_THRESHOLD:
+ features[f_j], features[n_total_constants] = features[n_total_constants], features[f_j]
- n_found_constants += 1
- n_total_constants += 1
+ n_found_constants += 1
+ n_total_constants += 1
+ continue
- else:
- f_i -= 1
- features[f_i], features[f_j] = features[f_j], features[f_i]
+ f_i -= 1
+ features[f_i], features[f_j] = features[f_j], features[f_i]
- # Evaluate all splits
- self.criterion.reset()
- p = start
+ # Evaluate all splits
+ self.criterion.reset()
+ p = start
- while p < end:
- while (p + 1 < end and
- Xf[p + 1] <= Xf[p] + FEATURE_THRESHOLD):
- p += 1
+ while p < end:
+ while p + 1 < end and Xf[p + 1] <= Xf[p] + FEATURE_THRESHOLD:
+ p += 1
- # (p + 1 >= end) or (X[samples[p + 1], current.feature] >
- # X[samples[p], current.feature])
- p += 1
- # (p >= end) or (X[samples[p], current.feature] >
- # X[samples[p - 1], current.feature])
+ # (p + 1 >= end) or (X[samples[p + 1], current.feature] >
+ # X[samples[p], current.feature])
+ p += 1
+ # (p >= end) or (X[samples[p], current.feature] >
+ # X[samples[p - 1], current.feature])
+
+ if p >= end:
+ continue
- if p < end:
- current.pos = p
+ current.pos = p
- # Reject if min_samples_leaf is not guaranteed
- if (((current.pos - start) < min_samples_leaf) or
- ((end - current.pos) < min_samples_leaf)):
- continue
+ # Reject if min_samples_leaf is not guaranteed
+ if (((current.pos - start) < min_samples_leaf) or
+ ((end - current.pos) < min_samples_leaf)):
+ continue
- self.criterion.update(current.pos)
+ self.criterion.update(current.pos)
- # Reject if min_weight_leaf is not satisfied
- if ((self.criterion.weighted_n_left < min_weight_leaf) or
- (self.criterion.weighted_n_right < min_weight_leaf)):
- continue
+ # Reject if min_weight_leaf is not satisfied
+ if ((self.criterion.weighted_n_left < min_weight_leaf) or
+ (self.criterion.weighted_n_right < min_weight_leaf)):
+ continue
- current_proxy_improvement = self.criterion.proxy_impurity_improvement()
+ current_proxy_improvement = self.criterion.proxy_impurity_improvement()
- if current_proxy_improvement > best_proxy_improvement:
- best_proxy_improvement = current_proxy_improvement
- # sum of halves is used to avoid infinite value
- current.threshold = Xf[p - 1] / 2.0 + Xf[p] / 2.0
+ if current_proxy_improvement > best_proxy_improvement:
+ best_proxy_improvement = current_proxy_improvement
+ # sum of halves is used to avoid infinite value
+ current.threshold = Xf[p - 1] / 2.0 + Xf[p] / 2.0
- if ((current.threshold == Xf[p]) or
- (current.threshold == INFINITY) or
- (current.threshold == -INFINITY)):
- current.threshold = Xf[p - 1]
+ if (
+ current.threshold == Xf[p] or
+ current.threshold == INFINITY or
+ current.threshold == -INFINITY
+ ):
+ current.threshold = Xf[p - 1]
- best = current # copy
+ best = current # copy
# Reorganize into samples[start:best.pos] + samples[best.pos:end]
if best.pos < end:
@@ -438,11 +424,11 @@ cdef class BestSplitter(BaseDenseSplitter):
# Respect invariant for constant features: the original order of
# element in features[:n_known_constants] must be preserved for sibling
# and child nodes
- memcpy(features, constant_features, sizeof(SIZE_t) * n_known_constants)
+ memcpy(&features[0], &constant_features[0], sizeof(SIZE_t) * n_known_constants)
# Copy newly found constant features
- memcpy(constant_features + n_known_constants,
- features + n_known_constants,
+ memcpy(&constant_features[n_known_constants],
+ &features[n_known_constants],
sizeof(SIZE_t) * n_found_constants)
# Return values
@@ -582,15 +568,15 @@ cdef class RandomSplitter(BaseDenseSplitter):
or 0 otherwise.
"""
# Draw random splits and pick the best
- cdef SIZE_t* samples = self.samples
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t start = self.start
cdef SIZE_t end = self.end
- cdef SIZE_t* features = self.features
- cdef SIZE_t* constant_features = self.constant_features
+ cdef SIZE_t[::1] features = self.features
+ cdef SIZE_t[::1] constant_features = self.constant_features
cdef SIZE_t n_features = self.n_features
- cdef DTYPE_t* Xf = self.feature_values
+ cdef DTYPE_t[::1] Xf = self.feature_values
cdef SIZE_t max_features = self.max_features
cdef SIZE_t min_samples_leaf = self.min_samples_leaf
cdef double min_weight_leaf = self.min_weight_leaf
@@ -654,78 +640,78 @@ cdef class RandomSplitter(BaseDenseSplitter):
# f_j in the interval [n_drawn_constants, n_known_constants[
features[n_drawn_constants], features[f_j] = features[f_j], features[n_drawn_constants]
n_drawn_constants += 1
+ continue
- else:
- # f_j in the interval [n_known_constants, f_i - n_found_constants[
- f_j += n_found_constants
- # f_j in the interval [n_total_constants, f_i[
+ # f_j in the interval [n_known_constants, f_i - n_found_constants[
+ f_j += n_found_constants
+ # f_j in the interval [n_total_constants, f_i[
- current.feature = features[f_j]
+ current.feature = features[f_j]
- # Find min, max
- min_feature_value = self.X[samples[start], current.feature]
- max_feature_value = min_feature_value
- Xf[start] = min_feature_value
+ # Find min, max
+ min_feature_value = self.X[samples[start], current.feature]
+ max_feature_value = min_feature_value
+ Xf[start] = min_feature_value
- for p in range(start + 1, end):
- current_feature_value = self.X[samples[p], current.feature]
- Xf[p] = current_feature_value
+ for p in range(start + 1, end):
+ current_feature_value = self.X[samples[p], current.feature]
+ Xf[p] = current_feature_value
- if current_feature_value < min_feature_value:
- min_feature_value = current_feature_value
- elif current_feature_value > max_feature_value:
- max_feature_value = current_feature_value
+ if current_feature_value < min_feature_value:
+ min_feature_value = current_feature_value
+ elif current_feature_value > max_feature_value:
+ max_feature_value = current_feature_value
- if max_feature_value <= min_feature_value + FEATURE_THRESHOLD:
- features[f_j], features[n_total_constants] = features[n_total_constants], current.feature
+ if max_feature_value <= min_feature_value + FEATURE_THRESHOLD:
+ features[f_j], features[n_total_constants] = features[n_total_constants], current.feature
- n_found_constants += 1
- n_total_constants += 1
+ n_found_constants += 1
+ n_total_constants += 1
+ continue
- else:
- f_i -= 1
- features[f_i], features[f_j] = features[f_j], features[f_i]
+ f_i -= 1
+ features[f_i], features[f_j] = features[f_j], features[f_i]
- # Draw a random threshold
- current.threshold = rand_uniform(min_feature_value,
- max_feature_value,
- random_state)
+ # Draw a random threshold
+ current.threshold = rand_uniform(min_feature_value,
+ max_feature_value,
+ random_state)
- if current.threshold == max_feature_value:
- current.threshold = min_feature_value
+ if current.threshold == max_feature_value:
+ current.threshold = min_feature_value
- # Partition
- p, partition_end = start, end
- while p < partition_end:
- if Xf[p] <= current.threshold:
- p += 1
- else:
- partition_end -= 1
+ # Partition
+ p, partition_end = start, end
+ while p < partition_end:
+ if Xf[p] <= current.threshold:
+ p += 1
+ else:
+ partition_end -= 1
- Xf[p], Xf[partition_end] = Xf[partition_end], Xf[p]
- samples[p], samples[partition_end] = samples[partition_end], samples[p]
+ Xf[p], Xf[partition_end] = Xf[partition_end], Xf[p]
+ samples[p], samples[partition_end] = samples[partition_end], samples[p]
- current.pos = partition_end
+ current.pos = partition_end
- # Reject if min_samples_leaf is not guaranteed
- if (((current.pos - start) < min_samples_leaf) or
- ((end - current.pos) < min_samples_leaf)):
- continue
+ # Reject if min_samples_leaf is not guaranteed
+ if (((current.pos - start) < min_samples_leaf) or
+ ((end - current.pos) < min_samples_leaf)):
+ continue
- # Evaluate split
- self.criterion.reset()
- self.criterion.update(current.pos)
+ # Evaluate split
+ self.criterion.reset()
+ self.criterion.update(current.pos)
- # Reject if min_weight_leaf is not satisfied
- if ((self.criterion.weighted_n_left < min_weight_leaf) or
- (self.criterion.weighted_n_right < min_weight_leaf)):
- continue
+ # Reject if min_weight_leaf is not satisfied
+ if ((self.criterion.weighted_n_left < min_weight_leaf) or
+ (self.criterion.weighted_n_right < min_weight_leaf)):
+ continue
- current_proxy_improvement = self.criterion.proxy_impurity_improvement()
+ current_proxy_improvement = self.criterion.proxy_impurity_improvement()
- if current_proxy_improvement > best_proxy_improvement:
- best_proxy_improvement = current_proxy_improvement
- best = current # copy
+ if current_proxy_improvement > best_proxy_improvement:
+ best_proxy_improvement = current_proxy_improvement
+ best = current # copy
# Reorganize into samples[start:best.pos] + samples[best.pos:end]
if best.pos < end:
@@ -750,11 +736,11 @@ cdef class RandomSplitter(BaseDenseSplitter):
# Respect invariant for constant features: the original order of
# element in features[:n_known_constants] must be preserved for sibling
# and child nodes
- memcpy(features, constant_features, sizeof(SIZE_t) * n_known_constants)
+ memcpy(&features[0], &constant_features[0], sizeof(SIZE_t) * n_known_constants)
# Copy newly found constant features
- memcpy(constant_features + n_known_constants,
- features + n_known_constants,
+ memcpy(&constant_features[n_known_constants],
+ &features[n_known_constants],
sizeof(SIZE_t) * n_found_constants)
# Return values
@@ -765,34 +751,21 @@ cdef class RandomSplitter(BaseDenseSplitter):
cdef class BaseSparseSplitter(Splitter):
# The sparse splitter works only with csc sparse matrix format
- cdef DTYPE_t* X_data
- cdef INT32_t* X_indices
- cdef INT32_t* X_indptr
+ cdef DTYPE_t[::1] X_data
+ cdef INT32_t[::1] X_indices
+ cdef INT32_t[::1] X_indptr
cdef SIZE_t n_total_samples
- cdef SIZE_t* index_to_samples
- cdef SIZE_t* sorted_samples
+ cdef SIZE_t[::1] index_to_samples
+ cdef SIZE_t[::1] sorted_samples
def __cinit__(self, Criterion criterion, SIZE_t max_features,
SIZE_t min_samples_leaf, double min_weight_leaf,
object random_state):
# Parent __cinit__ is automatically called
-
- self.X_data = NULL
- self.X_indices = NULL
- self.X_indptr = NULL
-
self.n_total_samples = 0
- self.index_to_samples = NULL
- self.sorted_samples = NULL
-
- def __dealloc__(self):
- """Deallocate memory."""
- free(self.index_to_samples)
- free(self.sorted_samples)
-
cdef int init(self,
object X,
const DOUBLE_t[:, ::1] y,
@@ -808,31 +781,24 @@ cdef class BaseSparseSplitter(Splitter):
if not isinstance(X, csc_matrix):
raise ValueError("X should be in csc format")
- cdef SIZE_t* samples = self.samples
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t n_samples = self.n_samples
# Initialize X
- cdef np.ndarray[dtype=DTYPE_t, ndim=1] data = X.data
- cdef np.ndarray[dtype=INT32_t, ndim=1] indices = X.indices
- cdef np.ndarray[dtype=INT32_t, ndim=1] indptr = X.indptr
cdef SIZE_t n_total_samples = X.shape[0]
- self.X_data = data.data
- self.X_indices = indices.data
- self.X_indptr = indptr.data
+ self.X_data = X.data
+ self.X_indices = X.indices
+ self.X_indptr = X.indptr
self.n_total_samples = n_total_samples
# Initialize auxiliary array used to perform split
- safe_realloc(&self.index_to_samples, n_total_samples)
- safe_realloc(&self.sorted_samples, n_samples)
+ self.index_to_samples = np.full(n_total_samples, fill_value=-1, dtype=np.intp)
+ self.sorted_samples = np.empty(n_samples, dtype=np.intp)
- cdef SIZE_t* index_to_samples = self.index_to_samples
cdef SIZE_t p
- for p in range(n_total_samples):
- index_to_samples[p] = -1
-
for p in range(n_samples):
- index_to_samples[samples[p]] = p
+ self.index_to_samples[samples[p]] = p
return 0
cdef inline SIZE_t _partition(self, double threshold,
@@ -843,9 +809,9 @@ cdef class BaseSparseSplitter(Splitter):
cdef SIZE_t p
cdef SIZE_t partition_end
- cdef DTYPE_t* Xf = self.feature_values
- cdef SIZE_t* samples = self.samples
- cdef SIZE_t* index_to_samples = self.index_to_samples
+ cdef DTYPE_t[::1] Xf = self.feature_values
+ cdef SIZE_t[::1] samples = self.samples
+ cdef SIZE_t[::1] index_to_samples = self.index_to_samples
if threshold < 0.:
p = self.start
@@ -905,6 +871,12 @@ cdef class BaseSparseSplitter(Splitter):
cdef SIZE_t indptr_end = self.X_indptr[feature + 1]
cdef SIZE_t n_indices = (indptr_end - indptr_start)
cdef SIZE_t n_samples = self.end - self.start
+ cdef SIZE_t[::1] samples = self.samples
+ cdef DTYPE_t[::1] feature_values = self.feature_values
+ cdef SIZE_t[::1] index_to_samples = self.index_to_samples
+ cdef SIZE_t[::1] sorted_samples = self.sorted_samples
+ cdef INT32_t[::1] X_indices = self.X_indices
+ cdef DTYPE_t[::1] X_data = self.X_data
# Use binary search if n_samples * log(n_indices) <
# n_indices and index_to_samples approach otherwise.
@@ -913,22 +885,22 @@ cdef class BaseSparseSplitter(Splitter):
# approach.
if ((1 - is_samples_sorted[0]) * n_samples * log(n_samples) +
n_samples * log(n_indices) < EXTRACT_NNZ_SWITCH * n_indices):
- extract_nnz_binary_search(self.X_indices, self.X_data,
+ extract_nnz_binary_search(X_indices, X_data,
indptr_start, indptr_end,
- self.samples, self.start, self.end,
- self.index_to_samples,
- self.feature_values,
+ samples, self.start, self.end,
+ index_to_samples,
+ feature_values,
end_negative, start_positive,
- self.sorted_samples, is_samples_sorted)
+ sorted_samples, is_samples_sorted)
# Using an index to samples technique to extract non zero values
# index_to_samples is a mapping from X_indices to samples
else:
- extract_nnz_index_to_samples(self.X_indices, self.X_data,
+ extract_nnz_index_to_samples(X_indices, X_data,
indptr_start, indptr_end,
- self.samples, self.start, self.end,
- self.index_to_samples,
- self.feature_values,
+ samples, self.start, self.end,
+ index_to_samples,
+ feature_values,
end_negative, start_positive)
@@ -937,7 +909,7 @@ cdef int compare_SIZE_t(const void* a, const void* b) nogil:
return ((a)[0] - (b)[0])
-cdef inline void binary_search(INT32_t* sorted_array,
+cdef inline void binary_search(INT32_t[::1] sorted_array,
INT32_t start, INT32_t end,
SIZE_t value, SIZE_t* index,
INT32_t* new_start) nogil:
@@ -962,15 +934,15 @@ cdef inline void binary_search(INT32_t* sorted_array,
new_start[0] = start
-cdef inline void extract_nnz_index_to_samples(INT32_t* X_indices,
- DTYPE_t* X_data,
+cdef inline void extract_nnz_index_to_samples(INT32_t[::1] X_indices,
+ DTYPE_t[::1] X_data,
INT32_t indptr_start,
INT32_t indptr_end,
- SIZE_t* samples,
+ SIZE_t[::1] samples,
SIZE_t start,
SIZE_t end,
- SIZE_t* index_to_samples,
- DTYPE_t* Xf,
+ SIZE_t[::1] index_to_samples,
+ DTYPE_t[::1] Xf,
SIZE_t* end_negative,
SIZE_t* start_positive) nogil:
"""Extract and partition values for a feature using index_to_samples.
@@ -1002,18 +974,18 @@ cdef inline void extract_nnz_index_to_samples(INT32_t* X_indices,
start_positive[0] = start_positive_
-cdef inline void extract_nnz_binary_search(INT32_t* X_indices,
- DTYPE_t* X_data,
+cdef inline void extract_nnz_binary_search(INT32_t[::1] X_indices,
+ DTYPE_t[::1] X_data,
INT32_t indptr_start,
INT32_t indptr_end,
- SIZE_t* samples,
+ SIZE_t[::1] samples,
SIZE_t start,
SIZE_t end,
- SIZE_t* index_to_samples,
- DTYPE_t* Xf,
+ SIZE_t[::1] index_to_samples,
+ DTYPE_t[::1] Xf,
SIZE_t* end_negative,
SIZE_t* start_positive,
- SIZE_t* sorted_samples,
+ SIZE_t[::1] sorted_samples,
bint* is_samples_sorted) nogil:
"""Extract and partition values for a given feature using binary search.
@@ -1027,9 +999,9 @@ cdef inline void extract_nnz_binary_search(INT32_t* X_indices,
if not is_samples_sorted[0]:
n_samples = end - start
- memcpy(sorted_samples + start, samples + start,
+ memcpy(&sorted_samples[start], &samples[start],
n_samples * sizeof(SIZE_t))
- qsort(sorted_samples + start, n_samples, sizeof(SIZE_t),
+ qsort(&sorted_samples[start], n_samples, sizeof(SIZE_t),
compare_SIZE_t)
is_samples_sorted[0] = 1
@@ -1074,7 +1046,7 @@ cdef inline void extract_nnz_binary_search(INT32_t* X_indices,
start_positive[0] = start_positive_
-cdef inline void sparse_swap(SIZE_t* index_to_samples, SIZE_t* samples,
+cdef inline void sparse_swap(SIZE_t[::1] index_to_samples, SIZE_t[::1] samples,
SIZE_t pos_1, SIZE_t pos_2) nogil:
"""Swap sample pos_1 and pos_2 preserving sparse invariant."""
samples[pos_1], samples[pos_2] = samples[pos_2], samples[pos_1]
@@ -1100,21 +1072,16 @@ cdef class BestSparseSplitter(BaseSparseSplitter):
or 0 otherwise.
"""
# Find the best split
- cdef SIZE_t* samples = self.samples
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t start = self.start
cdef SIZE_t end = self.end
- cdef INT32_t* X_indices = self.X_indices
- cdef INT32_t* X_indptr = self.X_indptr
- cdef DTYPE_t* X_data = self.X_data
-
- cdef SIZE_t* features = self.features
- cdef SIZE_t* constant_features = self.constant_features
+ cdef SIZE_t[::1] features = self.features
+ cdef SIZE_t[::1] constant_features = self.constant_features
cdef SIZE_t n_features = self.n_features
- cdef DTYPE_t* Xf = self.feature_values
- cdef SIZE_t* sorted_samples = self.sorted_samples
- cdef SIZE_t* index_to_samples = self.index_to_samples
+ cdef DTYPE_t[::1] Xf = self.feature_values
+ cdef SIZE_t[::1] index_to_samples = self.index_to_samples
cdef SIZE_t max_features = self.max_features
cdef SIZE_t min_samples_leaf = self.min_samples_leaf
cdef double min_weight_leaf = self.min_weight_leaf
@@ -1184,101 +1151,104 @@ cdef class BestSparseSplitter(BaseSparseSplitter):
features[f_j], features[n_drawn_constants] = features[n_drawn_constants], features[f_j]
n_drawn_constants += 1
+ continue
- else:
- # f_j in the interval [n_known_constants, f_i - n_found_constants[
- f_j += n_found_constants
- # f_j in the interval [n_total_constants, f_i[
-
- current.feature = features[f_j]
- self.extract_nnz(current.feature, &end_negative, &start_positive,
- &is_samples_sorted)
- # Sort the positive and negative parts of `Xf`
- sort(Xf + start, samples + start, end_negative - start)
- if start_positive < end:
- sort(Xf + start_positive, samples + start_positive,
- end - start_positive)
-
- # Update index_to_samples to take into account the sort
- for p in range(start, end_negative):
- index_to_samples[samples[p]] = p
- for p in range(start_positive, end):
- index_to_samples[samples[p]] = p
-
- # Add one or two zeros in Xf, if there is any
- if end_negative < start_positive:
- start_positive -= 1
- Xf[start_positive] = 0.
-
- if end_negative != start_positive:
- Xf[end_negative] = 0.
- end_negative += 1
-
- if Xf[end - 1] <= Xf[start] + FEATURE_THRESHOLD:
- features[f_j], features[n_total_constants] = features[n_total_constants], features[f_j]
-
- n_found_constants += 1
- n_total_constants += 1
+ # f_j in the interval [n_known_constants, f_i - n_found_constants[
+ f_j += n_found_constants
+ # f_j in the interval [n_total_constants, f_i[
- else:
- f_i -= 1
- features[f_i], features[f_j] = features[f_j], features[f_i]
-
- # Evaluate all splits
- self.criterion.reset()
- p = start
+ current.feature = features[f_j]
+ self.extract_nnz(current.feature, &end_negative, &start_positive,
+ &is_samples_sorted)
+ # Sort the positive and negative parts of `Xf`
+ sort(&Xf[start], &samples[start], end_negative - start)
+ if start_positive < end:
+ sort(&Xf[start_positive], &samples[start_positive],
+ end - start_positive)
+
+ # Update index_to_samples to take into account the sort
+ for p in range(start, end_negative):
+ index_to_samples[samples[p]] = p
+ for p in range(start_positive, end):
+ index_to_samples[samples[p]] = p
+
+ # Add one or two zeros in Xf, if there is any
+ if end_negative < start_positive:
+ start_positive -= 1
+ Xf[start_positive] = 0.
+
+ if end_negative != start_positive:
+ Xf[end_negative] = 0.
+ end_negative += 1
+
+ if Xf[end - 1] <= Xf[start] + FEATURE_THRESHOLD:
+ features[f_j], features[n_total_constants] = features[n_total_constants], features[f_j]
+
+ n_found_constants += 1
+ n_total_constants += 1
+ continue
+
+ f_i -= 1
+ features[f_i], features[f_j] = features[f_j], features[f_i]
+
+ # Evaluate all splits
+ self.criterion.reset()
+ p = start
- while p < end:
- if p + 1 != end_negative:
- p_next = p + 1
- else:
- p_next = start_positive
+ while p < end:
+ if p + 1 != end_negative:
+ p_next = p + 1
+ else:
+ p_next = start_positive
- while (p_next < end and
- Xf[p_next] <= Xf[p] + FEATURE_THRESHOLD):
- p = p_next
- if p + 1 != end_negative:
- p_next = p + 1
- else:
- p_next = start_positive
+ while (p_next < end and
+ Xf[p_next] <= Xf[p] + FEATURE_THRESHOLD):
+ p = p_next
+ if p + 1 != end_negative:
+ p_next = p + 1
+ else:
+ p_next = start_positive
- # (p_next >= end) or (X[samples[p_next], current.feature] >
- # X[samples[p], current.feature])
- p_prev = p
- p = p_next
- # (p >= end) or (X[samples[p], current.feature] >
- # X[samples[p_prev], current.feature])
+ # (p_next >= end) or (X[samples[p_next], current.feature] >
+ # X[samples[p], current.feature])
+ p_prev = p
+ p = p_next
+ # (p >= end) or (X[samples[p], current.feature] >
+ # X[samples[p_prev], current.feature])
+ if p >= end:
+ continue
- if p < end:
- current.pos = p
+ current.pos = p
- # Reject if min_samples_leaf is not guaranteed
- if (((current.pos - start) < min_samples_leaf) or
- ((end - current.pos) < min_samples_leaf)):
- continue
+ # Reject if min_samples_leaf is not guaranteed
+ if (((current.pos - start) < min_samples_leaf) or
+ ((end - current.pos) < min_samples_leaf)):
+ continue
- self.criterion.update(current.pos)
+ self.criterion.update(current.pos)
- # Reject if min_weight_leaf is not satisfied
- if ((self.criterion.weighted_n_left < min_weight_leaf) or
- (self.criterion.weighted_n_right < min_weight_leaf)):
- continue
+ # Reject if min_weight_leaf is not satisfied
+ if ((self.criterion.weighted_n_left < min_weight_leaf) or
+ (self.criterion.weighted_n_right < min_weight_leaf)):
+ continue
- current_proxy_improvement = self.criterion.proxy_impurity_improvement()
+ current_proxy_improvement = self.criterion.proxy_impurity_improvement()
- if current_proxy_improvement > best_proxy_improvement:
- best_proxy_improvement = current_proxy_improvement
- # sum of halves used to avoid infinite values
- current.threshold = Xf[p_prev] / 2.0 + Xf[p] / 2.0
+ if current_proxy_improvement > best_proxy_improvement:
+ best_proxy_improvement = current_proxy_improvement
+ # sum of halves used to avoid infinite values
+ current.threshold = Xf[p_prev] / 2.0 + Xf[p] / 2.0
- if ((current.threshold == Xf[p]) or
- (current.threshold == INFINITY) or
- (current.threshold == -INFINITY)):
- current.threshold = Xf[p_prev]
+ if (
+ current.threshold == Xf[p] or
+ current.threshold == INFINITY or
+ current.threshold == -INFINITY
+ ):
+ current.threshold = Xf[p_prev]
- best = current
+ best = current
# Reorganize into samples[start:best.pos] + samples[best.pos:end]
if best.pos < end:
@@ -1298,11 +1268,11 @@ cdef class BestSparseSplitter(BaseSparseSplitter):
# Respect invariant for constant features: the original order of
# element in features[:n_known_constants] must be preserved for sibling
# and child nodes
- memcpy(features, constant_features, sizeof(SIZE_t) * n_known_constants)
+ memcpy(&features[0], &constant_features[0], sizeof(SIZE_t) * n_known_constants)
# Copy newly found constant features
- memcpy(constant_features + n_known_constants,
- features + n_known_constants,
+ memcpy(&constant_features[n_known_constants],
+ &features[n_known_constants],
sizeof(SIZE_t) * n_found_constants)
# Return values
@@ -1329,21 +1299,16 @@ cdef class RandomSparseSplitter(BaseSparseSplitter):
or 0 otherwise.
"""
# Find the best split
- cdef SIZE_t* samples = self.samples
+ cdef SIZE_t[::1] samples = self.samples
cdef SIZE_t start = self.start
cdef SIZE_t end = self.end
- cdef INT32_t* X_indices = self.X_indices
- cdef INT32_t* X_indptr = self.X_indptr
- cdef DTYPE_t* X_data = self.X_data
-
- cdef SIZE_t* features = self.features
- cdef SIZE_t* constant_features = self.constant_features
+ cdef SIZE_t[::1] features = self.features
+ cdef SIZE_t[::1] constant_features = self.constant_features
cdef SIZE_t n_features = self.n_features
- cdef DTYPE_t* Xf = self.feature_values
- cdef SIZE_t* sorted_samples = self.sorted_samples
- cdef SIZE_t* index_to_samples = self.index_to_samples
+ cdef DTYPE_t[::1] Xf = self.feature_values
+ cdef SIZE_t[::1] index_to_samples = self.index_to_samples
cdef SIZE_t max_features = self.max_features
cdef SIZE_t min_samples_leaf = self.min_samples_leaf
cdef double min_weight_leaf = self.min_weight_leaf
@@ -1416,97 +1381,92 @@ cdef class RandomSparseSplitter(BaseSparseSplitter):
features[f_j], features[n_drawn_constants] = features[n_drawn_constants], features[f_j]
n_drawn_constants += 1
+ continue
+
+ # f_j in the interval [n_known_constants, f_i - n_found_constants[
+ f_j += n_found_constants
+ # f_j in the interval [n_total_constants, f_i[
+
+ current.feature = features[f_j]
+ self.extract_nnz(current.feature,
+ &end_negative, &start_positive,
+ &is_samples_sorted)
+
+ if end_negative != start_positive:
+ # There is a zero
+ min_feature_value = 0
+ max_feature_value = 0
else:
- # f_j in the interval [n_known_constants, f_i - n_found_constants[
- f_j += n_found_constants
- # f_j in the interval [n_total_constants, f_i[
+ min_feature_value = Xf[start]
+ max_feature_value = min_feature_value
- current.feature = features[f_j]
+ # Find min, max in Xf[start:end_negative]
+ for p in range(start, end_negative):
+ current_feature_value = Xf[p]
- self.extract_nnz(current.feature,
- &end_negative, &start_positive,
- &is_samples_sorted)
+ if current_feature_value < min_feature_value:
+ min_feature_value = current_feature_value
+ elif current_feature_value > max_feature_value:
+ max_feature_value = current_feature_value
- # Add one or two zeros in Xf, if there is any
- if end_negative < start_positive:
- start_positive -= 1
- Xf[start_positive] = 0.
+ # Update min, max given Xf[start_positive:end]
+ for p in range(start_positive, end):
+ current_feature_value = Xf[p]
- if end_negative != start_positive:
- Xf[end_negative] = 0.
- end_negative += 1
+ if current_feature_value < min_feature_value:
+ min_feature_value = current_feature_value
+ elif current_feature_value > max_feature_value:
+ max_feature_value = current_feature_value
- # Find min, max in Xf[start:end_negative]
- min_feature_value = Xf[start]
- max_feature_value = min_feature_value
+ if max_feature_value <= min_feature_value + FEATURE_THRESHOLD:
+ features[f_j] = features[n_total_constants]
+ features[n_total_constants] = current.feature
- for p in range(start, end_negative):
- current_feature_value = Xf[p]
+ n_found_constants += 1
+ n_total_constants += 1
+ continue
- if current_feature_value < min_feature_value:
- min_feature_value = current_feature_value
- elif current_feature_value > max_feature_value:
- max_feature_value = current_feature_value
+ f_i -= 1
+ features[f_i], features[f_j] = features[f_j], features[f_i]
- # Update min, max given Xf[start_positive:end]
- for p in range(start_positive, end):
- current_feature_value = Xf[p]
+ # Draw a random threshold
+ current.threshold = rand_uniform(min_feature_value,
+ max_feature_value,
+ random_state)
- if current_feature_value < min_feature_value:
- min_feature_value = current_feature_value
- elif current_feature_value > max_feature_value:
- max_feature_value = current_feature_value
+ if current.threshold == max_feature_value:
+ current.threshold = min_feature_value
- if max_feature_value <= min_feature_value + FEATURE_THRESHOLD:
- features[f_j] = features[n_total_constants]
- features[n_total_constants] = current.feature
+ # Partition
+ current.pos = self._partition(current.threshold,
+ end_negative,
+ start_positive,
+ start_positive)
- n_found_constants += 1
- n_total_constants += 1
+ # Reject if min_samples_leaf is not guaranteed
+ if (((current.pos - start) < min_samples_leaf) or
+ ((end - current.pos) < min_samples_leaf)):
+ continue
- else:
- f_i -= 1
- features[f_i], features[f_j] = features[f_j], features[f_i]
-
- # Draw a random threshold
- current.threshold = rand_uniform(min_feature_value,
- max_feature_value,
- random_state)
-
- if current.threshold == max_feature_value:
- current.threshold = min_feature_value
-
- # Partition
- current.pos = self._partition(current.threshold,
- end_negative,
- start_positive,
- start_positive +
- (Xf[start_positive] == 0.))
-
- # Reject if min_samples_leaf is not guaranteed
- if (((current.pos - start) < min_samples_leaf) or
- ((end - current.pos) < min_samples_leaf)):
- continue
-
- # Evaluate split
- self.criterion.reset()
- self.criterion.update(current.pos)
-
- # Reject if min_weight_leaf is not satisfied
- if ((self.criterion.weighted_n_left < min_weight_leaf) or
- (self.criterion.weighted_n_right < min_weight_leaf)):
- continue
-
- current_proxy_improvement = self.criterion.proxy_impurity_improvement()
-
- if current_proxy_improvement > best_proxy_improvement:
- best_proxy_improvement = current_proxy_improvement
- self.criterion.children_impurity(¤t.impurity_left,
- ¤t.impurity_right)
- current.improvement = self.criterion.impurity_improvement(
- impurity, current.impurity_left, current.impurity_right)
- best = current
+ # Evaluate split
+ self.criterion.reset()
+ self.criterion.update(current.pos)
+
+ # Reject if min_weight_leaf is not satisfied
+ if ((self.criterion.weighted_n_left < min_weight_leaf) or
+ (self.criterion.weighted_n_right < min_weight_leaf)):
+ continue
+
+ current_proxy_improvement = self.criterion.proxy_impurity_improvement()
+
+ if current_proxy_improvement > best_proxy_improvement:
+ best_proxy_improvement = current_proxy_improvement
+ self.criterion.children_impurity(¤t.impurity_left,
+ ¤t.impurity_right)
+ current.improvement = self.criterion.impurity_improvement(
+ impurity, current.impurity_left, current.impurity_right)
+ best = current
# Reorganize into samples[start:best.pos] + samples[best.pos:end]
if best.pos < end:
@@ -1527,11 +1487,11 @@ cdef class RandomSparseSplitter(BaseSparseSplitter):
# Respect invariant for constant features: the original order of
# element in features[:n_known_constants] must be preserved for sibling
# and child nodes
- memcpy(features, constant_features, sizeof(SIZE_t) * n_known_constants)
+ memcpy(&features[0], &constant_features[0], sizeof(SIZE_t) * n_known_constants)
# Copy newly found constant features
- memcpy(constant_features + n_known_constants,
- features + n_known_constants,
+ memcpy(&constant_features[n_known_constants],
+ &features[n_known_constants],
sizeof(SIZE_t) * n_found_constants)
# Return values
diff --git a/sklearn/tree/_tree.pxd b/sklearn/tree/_tree.pxd
index 0874187ee98ae..55895a8279828 100644
--- a/sklearn/tree/_tree.pxd
+++ b/sklearn/tree/_tree.pxd
@@ -11,13 +11,13 @@
# See _tree.pyx for details.
import numpy as np
-cimport numpy as np
+cimport numpy as cnp
-ctypedef np.npy_float32 DTYPE_t # Type of X
-ctypedef np.npy_float64 DOUBLE_t # Type of y, sample_weight
-ctypedef np.npy_intp SIZE_t # Type for indices and counters
-ctypedef np.npy_int32 INT32_t # Signed 32 bit integer
-ctypedef np.npy_uint32 UINT32_t # Unsigned 32 bit integer
+ctypedef cnp.npy_float32 DTYPE_t # Type of X
+ctypedef cnp.npy_float64 DOUBLE_t # Type of y, sample_weight
+ctypedef cnp.npy_intp SIZE_t # Type for indices and counters
+ctypedef cnp.npy_int32 INT32_t # Signed 32 bit integer
+ctypedef cnp.npy_uint32 UINT32_t # Unsigned 32 bit integer
from ._splitter cimport Splitter
from ._splitter cimport SplitRecord
@@ -62,14 +62,14 @@ cdef class Tree:
cdef int _resize(self, SIZE_t capacity) nogil except -1
cdef int _resize_c(self, SIZE_t capacity=*) nogil except -1
- cdef np.ndarray _get_value_ndarray(self)
- cdef np.ndarray _get_node_ndarray(self)
+ cdef cnp.ndarray _get_value_ndarray(self)
+ cdef cnp.ndarray _get_node_ndarray(self)
- cpdef np.ndarray predict(self, object X)
+ cpdef cnp.ndarray predict(self, object X)
- cpdef np.ndarray apply(self, object X)
- cdef np.ndarray _apply_dense(self, object X)
- cdef np.ndarray _apply_sparse_csr(self, object X)
+ cpdef cnp.ndarray apply(self, object X)
+ cdef cnp.ndarray _apply_dense(self, object X)
+ cdef cnp.ndarray _apply_sparse_csr(self, object X)
cpdef object decision_path(self, object X)
cdef object _decision_path_dense(self, object X)
@@ -98,6 +98,6 @@ cdef class TreeBuilder:
cdef SIZE_t max_depth # Maximal tree depth
cdef double min_impurity_decrease # Impurity threshold for early stopping
- cpdef build(self, Tree tree, object X, np.ndarray y,
- np.ndarray sample_weight=*)
- cdef _check_input(self, object X, np.ndarray y, np.ndarray sample_weight)
+ cpdef build(self, Tree tree, object X, cnp.ndarray y,
+ cnp.ndarray sample_weight=*)
+ cdef _check_input(self, object X, cnp.ndarray y, cnp.ndarray sample_weight)
diff --git a/sklearn/tree/_tree.pyx b/sklearn/tree/_tree.pyx
index 85c44b5eaf9b8..c2a2e80caaec6 100644
--- a/sklearn/tree/_tree.pyx
+++ b/sklearn/tree/_tree.pyx
@@ -27,8 +27,8 @@ from libcpp cimport bool
import struct
import numpy as np
-cimport numpy as np
-np.import_array()
+cimport numpy as cnp
+cnp.import_array()
from scipy.sparse import issparse
from scipy.sparse import csr_matrix
@@ -37,11 +37,11 @@ from ._utils cimport safe_realloc
from ._utils cimport sizet_ptr_to_ndarray
cdef extern from "numpy/arrayobject.h":
- object PyArray_NewFromDescr(PyTypeObject* subtype, np.dtype descr,
- int nd, np.npy_intp* dims,
- np.npy_intp* strides,
+ object PyArray_NewFromDescr(PyTypeObject* subtype, cnp.dtype descr,
+ int nd, cnp.npy_intp* dims,
+ cnp.npy_intp* strides,
void* data, int flags, object obj)
- int PyArray_SetBaseObject(np.ndarray arr, PyObject* obj)
+ int PyArray_SetBaseObject(cnp.ndarray arr, PyObject* obj)
cdef extern from "" namespace "std" nogil:
cdef cppclass stack[T]:
@@ -87,13 +87,13 @@ NODE_DTYPE = np.asarray((&dummy)).dtype
cdef class TreeBuilder:
"""Interface for different tree building strategies."""
- cpdef build(self, Tree tree, object X, np.ndarray y,
- np.ndarray sample_weight=None):
+ cpdef build(self, Tree tree, object X, cnp.ndarray y,
+ cnp.ndarray sample_weight=None):
"""Build a decision tree from the training set (X, y)."""
pass
- cdef inline _check_input(self, object X, np.ndarray y,
- np.ndarray sample_weight):
+ cdef inline _check_input(self, object X, cnp.ndarray y,
+ cnp.ndarray sample_weight):
"""Check input dtype, layout and format"""
if issparse(X):
X = X.tocsc()
@@ -145,8 +145,8 @@ cdef class DepthFirstTreeBuilder(TreeBuilder):
self.max_depth = max_depth
self.min_impurity_decrease = min_impurity_decrease
- cpdef build(self, Tree tree, object X, np.ndarray y,
- np.ndarray sample_weight=None):
+ cpdef build(self, Tree tree, object X, cnp.ndarray y,
+ cnp.ndarray sample_weight=None):
"""Build a decision tree from the training set (X, y)."""
# check input
@@ -341,8 +341,8 @@ cdef class BestFirstTreeBuilder(TreeBuilder):
self.max_leaf_nodes = max_leaf_nodes
self.min_impurity_decrease = min_impurity_decrease
- cpdef build(self, Tree tree, object X, np.ndarray y,
- np.ndarray sample_weight=None):
+ cpdef build(self, Tree tree, object X, cnp.ndarray y,
+ cnp.ndarray sample_weight=None):
"""Build a decision tree from the training set (X, y)."""
# check input
@@ -624,7 +624,7 @@ cdef class Tree:
def __get__(self):
return self._get_value_ndarray()[:self.node_count]
- def __cinit__(self, int n_features, np.ndarray n_classes, int n_outputs):
+ def __cinit__(self, int n_features, cnp.ndarray n_classes, int n_outputs):
"""Constructor."""
cdef SIZE_t dummy = 0
size_t_dtype = np.array(dummy).dtype
@@ -699,9 +699,9 @@ cdef class Tree:
self.capacity = node_ndarray.shape[0]
if self._resize_c(self.capacity) != 0:
raise MemoryError("resizing tree to %d" % self.capacity)
- nodes = memcpy(self.nodes, ( node_ndarray).data,
+ nodes = memcpy(self.nodes, ( node_ndarray).data,
self.capacity * sizeof(Node))
- value = memcpy(self.value, ( value_ndarray).data,
+ value = memcpy(self.value, ( value_ndarray).data,
self.capacity * self.value_stride * sizeof(double))
cdef int _resize(self, SIZE_t capacity) nogil except -1:
@@ -789,7 +789,7 @@ cdef class Tree:
return node_id
- cpdef np.ndarray predict(self, object X):
+ cpdef cnp.ndarray predict(self, object X):
"""Predict target for X."""
out = self._get_value_ndarray().take(self.apply(X), axis=0,
mode='clip')
@@ -797,14 +797,14 @@ cdef class Tree:
out = out.reshape(X.shape[0], self.max_n_classes)
return out
- cpdef np.ndarray apply(self, object X):
+ cpdef cnp.ndarray apply(self, object X):
"""Finds the terminal region (=leaf node) for each sample in X."""
if issparse(X):
return self._apply_sparse_csr(X)
else:
return self._apply_dense(X)
- cdef inline np.ndarray _apply_dense(self, object X):
+ cdef inline cnp.ndarray _apply_dense(self, object X):
"""Finds the terminal region (=leaf node) for each sample in X."""
# Check input
@@ -820,7 +820,7 @@ cdef class Tree:
cdef SIZE_t n_samples = X.shape[0]
# Initialize output
- cdef np.ndarray[SIZE_t] out = np.zeros((n_samples,), dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] out = np.zeros((n_samples,), dtype=np.intp)
cdef SIZE_t* out_ptr = out.data
# Initialize auxiliary data-structure
@@ -842,7 +842,7 @@ cdef class Tree:
return out
- cdef inline np.ndarray _apply_sparse_csr(self, object X):
+ cdef inline cnp.ndarray _apply_sparse_csr(self, object X):
"""Finds the terminal region (=leaf node) for each sample in sparse X.
"""
# Check input
@@ -854,9 +854,9 @@ cdef class Tree:
raise ValueError("X.dtype should be np.float32, got %s" % X.dtype)
# Extract input
- cdef np.ndarray[ndim=1, dtype=DTYPE_t] X_data_ndarray = X.data
- cdef np.ndarray[ndim=1, dtype=INT32_t] X_indices_ndarray = X.indices
- cdef np.ndarray[ndim=1, dtype=INT32_t] X_indptr_ndarray = X.indptr
+ cdef cnp.ndarray[ndim=1, dtype=DTYPE_t] X_data_ndarray = X.data
+ cdef cnp.ndarray[ndim=1, dtype=INT32_t] X_indices_ndarray = X.indices
+ cdef cnp.ndarray[ndim=1, dtype=INT32_t] X_indptr_ndarray = X.indptr
cdef DTYPE_t* X_data = X_data_ndarray.data
cdef INT32_t* X_indices = X_indices_ndarray.data
@@ -866,8 +866,8 @@ cdef class Tree:
cdef SIZE_t n_features = X.shape[1]
# Initialize output
- cdef np.ndarray[SIZE_t, ndim=1] out = np.zeros((n_samples,),
- dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t, ndim=1] out = np.zeros((n_samples,),
+ dtype=np.intp)
cdef SIZE_t* out_ptr = out.data
# Initialize auxiliary data-structure
@@ -940,12 +940,12 @@ cdef class Tree:
cdef SIZE_t n_samples = X.shape[0]
# Initialize output
- cdef np.ndarray[SIZE_t] indptr = np.zeros(n_samples + 1, dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] indptr = np.zeros(n_samples + 1, dtype=np.intp)
cdef SIZE_t* indptr_ptr = indptr.data
- cdef np.ndarray[SIZE_t] indices = np.zeros(n_samples *
- (1 + self.max_depth),
- dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] indices = np.zeros(n_samples *
+ (1 + self.max_depth),
+ dtype=np.intp)
cdef SIZE_t* indices_ptr = indices.data
# Initialize auxiliary data-structure
@@ -973,8 +973,8 @@ cdef class Tree:
indptr_ptr[i + 1] += 1
indices = indices[:indptr[n_samples]]
- cdef np.ndarray[SIZE_t] data = np.ones(shape=len(indices),
- dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] data = np.ones(shape=len(indices),
+ dtype=np.intp)
out = csr_matrix((data, indices, indptr),
shape=(n_samples, self.node_count))
@@ -992,9 +992,9 @@ cdef class Tree:
raise ValueError("X.dtype should be np.float32, got %s" % X.dtype)
# Extract input
- cdef np.ndarray[ndim=1, dtype=DTYPE_t] X_data_ndarray = X.data
- cdef np.ndarray[ndim=1, dtype=INT32_t] X_indices_ndarray = X.indices
- cdef np.ndarray[ndim=1, dtype=INT32_t] X_indptr_ndarray = X.indptr
+ cdef cnp.ndarray[ndim=1, dtype=DTYPE_t] X_data_ndarray = X.data
+ cdef cnp.ndarray[ndim=1, dtype=INT32_t] X_indices_ndarray = X.indices
+ cdef cnp.ndarray[ndim=1, dtype=INT32_t] X_indptr_ndarray = X.indptr
cdef DTYPE_t* X_data = X_data_ndarray.data
cdef INT32_t* X_indices = X_indices_ndarray.data
@@ -1004,12 +1004,12 @@ cdef class Tree:
cdef SIZE_t n_features = X.shape[1]
# Initialize output
- cdef np.ndarray[SIZE_t] indptr = np.zeros(n_samples + 1, dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] indptr = np.zeros(n_samples + 1, dtype=np.intp)
cdef SIZE_t* indptr_ptr = indptr.data
- cdef np.ndarray[SIZE_t] indices = np.zeros(n_samples *
- (1 + self.max_depth),
- dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] indices = np.zeros(n_samples *
+ (1 + self.max_depth),
+ dtype=np.intp)
cdef SIZE_t* indices_ptr = indices.data
# Initialize auxiliary data-structure
@@ -1065,8 +1065,8 @@ cdef class Tree:
free(feature_to_sample)
indices = indices[:indptr[n_samples]]
- cdef np.ndarray[SIZE_t] data = np.ones(shape=len(indices),
- dtype=np.intp)
+ cdef cnp.ndarray[SIZE_t] data = np.ones(shape=len(indices),
+ dtype=np.intp)
out = csr_matrix((data, indices, indptr),
shape=(n_samples, self.node_count))
@@ -1083,7 +1083,7 @@ cdef class Tree:
cdef double normalizer = 0.
- cdef np.ndarray[np.float64_t, ndim=1] importances
+ cdef cnp.ndarray[cnp.float64_t, ndim=1] importances
importances = np.zeros((self.n_features,))
cdef DOUBLE_t* importance_data = importances.data
@@ -1111,40 +1111,40 @@ cdef class Tree:
return importances
- cdef np.ndarray _get_value_ndarray(self):
+ cdef cnp.ndarray _get_value_ndarray(self):
"""Wraps value as a 3-d NumPy array.
The array keeps a reference to this Tree, which manages the underlying
memory.
"""
- cdef np.npy_intp shape[3]
- shape[0] = self.node_count
- shape[1] = self.n_outputs
- shape[2] = self.max_n_classes
- cdef np.ndarray arr
- arr = np.PyArray_SimpleNewFromData(3, shape, np.NPY_DOUBLE, self.value)
+ cdef cnp.npy_intp shape[3]
+ shape[0] = self.node_count
+ shape[1] = self.n_outputs
+ shape[2] = self.max_n_classes
+ cdef cnp.ndarray arr
+ arr = cnp.PyArray_SimpleNewFromData(3, shape, cnp.NPY_DOUBLE, self.value)
Py_INCREF(self)
if PyArray_SetBaseObject(arr, self) < 0:
raise ValueError("Can't initialize array.")
return arr
- cdef np.ndarray _get_node_ndarray(self):
+ cdef cnp.ndarray _get_node_ndarray(self):
"""Wraps nodes as a NumPy struct array.
The array keeps a reference to this Tree, which manages the underlying
memory. Individual fields are publicly accessible as properties of the
Tree.
"""
- cdef np.npy_intp shape[1]
- shape[0] = self.node_count
- cdef np.npy_intp strides[1]
+ cdef cnp.npy_intp shape[1]
+ shape[0] = self.node_count
+ cdef cnp.npy_intp strides[1]
strides[0] = sizeof(Node)
- cdef np.ndarray arr
+ cdef cnp.ndarray arr
Py_INCREF(NODE_DTYPE)
- arr = PyArray_NewFromDescr( np.ndarray,
- NODE_DTYPE, 1, shape,
+ arr = PyArray_NewFromDescr( cnp.ndarray,
+ NODE_DTYPE, 1, shape,
strides, self.nodes,
- np.NPY_DEFAULT, None)
+ cnp.NPY_DEFAULT, None)
Py_INCREF(self)
if PyArray_SetBaseObject(arr, self) < 0:
raise ValueError("Can't initialize array.")
@@ -1680,10 +1680,10 @@ def ccp_pruning_path(Tree orig_tree):
cdef:
UINT32_t total_items = path_finder.count
- np.ndarray ccp_alphas = np.empty(shape=total_items,
- dtype=np.float64)
- np.ndarray impurities = np.empty(shape=total_items,
- dtype=np.float64)
+ cnp.ndarray ccp_alphas = np.empty(shape=total_items,
+ dtype=np.float64)
+ cnp.ndarray impurities = np.empty(shape=total_items,
+ dtype=np.float64)
UINT32_t count = 0
while count < total_items:
diff --git a/sklearn/tree/_utils.pxd b/sklearn/tree/_utils.pxd
index fe4aca67d7b52..9d41b757d85dc 100644
--- a/sklearn/tree/_utils.pxd
+++ b/sklearn/tree/_utils.pxd
@@ -8,16 +8,15 @@
# See _utils.pyx for details.
-import numpy as np
-cimport numpy as np
+cimport numpy as cnp
from ._tree cimport Node
from ..neighbors._quad_tree cimport Cell
-ctypedef np.npy_float32 DTYPE_t # Type of X
-ctypedef np.npy_float64 DOUBLE_t # Type of y, sample_weight
-ctypedef np.npy_intp SIZE_t # Type for indices and counters
-ctypedef np.npy_int32 INT32_t # Signed 32 bit integer
-ctypedef np.npy_uint32 UINT32_t # Unsigned 32 bit integer
+ctypedef cnp.npy_float32 DTYPE_t # Type of X
+ctypedef cnp.npy_float64 DOUBLE_t # Type of y, sample_weight
+ctypedef cnp.npy_intp SIZE_t # Type for indices and counters
+ctypedef cnp.npy_int32 INT32_t # Signed 32 bit integer
+ctypedef cnp.npy_uint32 UINT32_t # Unsigned 32 bit integer
cdef enum:
@@ -47,7 +46,7 @@ ctypedef fused realloc_ptr:
cdef realloc_ptr safe_realloc(realloc_ptr* p, size_t nelems) nogil except *
-cdef np.ndarray sizet_ptr_to_ndarray(SIZE_t* data, SIZE_t size)
+cdef cnp.ndarray sizet_ptr_to_ndarray(SIZE_t* data, SIZE_t size)
cdef SIZE_t rand_int(SIZE_t low, SIZE_t high,
diff --git a/sklearn/tree/_utils.pyx b/sklearn/tree/_utils.pyx
index ba4c0f716a985..7346070b7a149 100644
--- a/sklearn/tree/_utils.pyx
+++ b/sklearn/tree/_utils.pyx
@@ -12,9 +12,8 @@ from libc.stdlib cimport malloc
from libc.stdlib cimport realloc
from libc.math cimport log as ln
-import numpy as np
-cimport numpy as np
-np.import_array()
+cimport numpy as cnp
+cnp.import_array()
from ..utils._random cimport our_rand_r
@@ -50,11 +49,11 @@ def _realloc_test():
assert False
-cdef inline np.ndarray sizet_ptr_to_ndarray(SIZE_t* data, SIZE_t size):
+cdef inline cnp.ndarray sizet_ptr_to_ndarray(SIZE_t* data, SIZE_t size):
"""Return copied data as 1D numpy array of intp's."""
- cdef np.npy_intp shape[1]
- shape[0] = size
- return np.PyArray_SimpleNewFromData(1, shape, np.NPY_INTP, data).copy()
+ cdef cnp.npy_intp shape[1]
+ shape[0] = size
+ return cnp.PyArray_SimpleNewFromData(1, shape, cnp.NPY_INTP, data).copy()
cdef inline SIZE_t rand_int(SIZE_t low, SIZE_t high,
diff --git a/sklearn/utils/__init__.py b/sklearn/utils/__init__.py
index aa056e92b3d12..a4857d62f7a38 100644
--- a/sklearn/utils/__init__.py
+++ b/sklearn/utils/__init__.py
@@ -473,6 +473,10 @@ def resample(*arrays, replace=True, n_samples=None, random_state=None, stratify=
Sequence of resampled copies of the collections. The original arrays
are not impacted.
+ See Also
+ --------
+ shuffle : Shuffle arrays or sparse matrices in a consistent way.
+
Examples
--------
It is possible to mix sparse and dense arrays in the same run::
@@ -512,10 +516,6 @@ def resample(*arrays, replace=True, n_samples=None, random_state=None, stratify=
>>> resample(y, n_samples=5, replace=False, stratify=y,
... random_state=0)
[1, 1, 1, 0, 1]
-
- See Also
- --------
- shuffle
"""
max_n_samples = n_samples
random_state = check_random_state(random_state)
diff --git a/sklearn/utils/_estimator_html_repr.py b/sklearn/utils/_estimator_html_repr.py
index f8911b5c38b08..e5291b6de3701 100644
--- a/sklearn/utils/_estimator_html_repr.py
+++ b/sklearn/utils/_estimator_html_repr.py
@@ -1,5 +1,4 @@
from contextlib import closing
-from contextlib import suppress
from io import StringIO
from string import Template
import html
@@ -103,8 +102,16 @@ def _write_label_html(
def _get_visual_block(estimator):
"""Generate information about how to display an estimator."""
- with suppress(AttributeError):
- return estimator._sk_visual_block_()
+ if hasattr(estimator, "_sk_visual_block_"):
+ try:
+ return estimator._sk_visual_block_()
+ except Exception:
+ return _VisualBlock(
+ "single",
+ estimator,
+ names=estimator.__class__.__name__,
+ name_details=str(estimator),
+ )
if isinstance(estimator, str):
return _VisualBlock(
diff --git a/sklearn/utils/_testing.py b/sklearn/utils/_testing.py
index 453f3437307a9..4851322197d7b 100644
--- a/sklearn/utils/_testing.py
+++ b/sklearn/utils/_testing.py
@@ -22,6 +22,7 @@
import re
import contextlib
from collections.abc import Iterable
+from collections.abc import Sequence
import scipy as sp
from functools import wraps
@@ -34,7 +35,7 @@
# WindowsError only exist on Windows
try:
- WindowsError
+ WindowsError # type: ignore
except NameError:
WindowsError = None
@@ -60,6 +61,7 @@
check_is_fitted,
check_X_y,
)
+from sklearn.utils.fixes import threadpool_info
__all__ = [
@@ -414,7 +416,6 @@ def assert_allclose(
If None, it is set based on the provided arrays' dtypes.
atol : float, optional, default=0.
Absolute tolerance.
- If None, it is set based on the provided arrays' dtypes.
equal_nan : bool, optional, default=True
If True, NaNs will compare equal.
err_msg : str, optional, default=''
@@ -603,6 +604,38 @@ def __exit__(self, exc_type, exc_val, exc_tb):
_delete_folder(self.temp_folder)
+def _create_memmap_backed_array(array, filename, mmap_mode):
+ # https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
+ fp = np.memmap(filename, dtype=array.dtype, mode="w+", shape=array.shape)
+ fp[:] = array[:] # write array to memmap array
+ fp.flush()
+ memmap_backed_array = np.memmap(
+ filename, dtype=array.dtype, mode=mmap_mode, shape=array.shape
+ )
+ return memmap_backed_array
+
+
+def _create_aligned_memmap_backed_arrays(data, mmap_mode, folder):
+ if isinstance(data, np.ndarray):
+ filename = op.join(folder, "data.dat")
+ return _create_memmap_backed_array(data, filename, mmap_mode)
+
+ if isinstance(data, Sequence) and all(
+ isinstance(each, np.ndarray) for each in data
+ ):
+ return [
+ _create_memmap_backed_array(
+ array, op.join(folder, f"data{index}.dat"), mmap_mode
+ )
+ for index, array in enumerate(data)
+ ]
+
+ raise ValueError(
+ "When creating aligned memmap-backed arrays, input must be a single array or a"
+ " sequence of arrays"
+ )
+
+
def create_memmap_backed_data(data, mmap_mode="r", return_folder=False, aligned=False):
"""
Parameters
@@ -617,18 +650,23 @@ def create_memmap_backed_data(data, mmap_mode="r", return_folder=False, aligned=
"""
temp_folder = tempfile.mkdtemp(prefix="sklearn_testing_")
atexit.register(functools.partial(_delete_folder, temp_folder, warn=True))
+ # OpenBLAS is known to segfault with unaligned data on the Prescott
+ # architecture so force aligned=True on Prescott. For more details, see:
+ # https://github.com/scipy/scipy/issues/14886
+ has_prescott_openblas = any(
+ True
+ for info in threadpool_info()
+ if info["internal_api"] == "openblas"
+ # Prudently assume Prescott might be the architecture if it is unknown.
+ and info.get("architecture", "prescott").lower() == "prescott"
+ )
+ if has_prescott_openblas:
+ aligned = True
+
if aligned:
- if isinstance(data, np.ndarray) and data.flags.aligned:
- # https://numpy.org/doc/stable/reference/generated/numpy.memmap.html
- filename = op.join(temp_folder, "data.dat")
- fp = np.memmap(filename, dtype=data.dtype, mode="w+", shape=data.shape)
- fp[:] = data[:] # write data to memmap array
- fp.flush()
- memmap_backed_data = np.memmap(
- filename, dtype=data.dtype, mode=mmap_mode, shape=data.shape
- )
- else:
- raise ValueError("If aligned=True, input must be a single numpy array.")
+ memmap_backed_data = _create_aligned_memmap_backed_arrays(
+ data, mmap_mode, temp_folder
+ )
else:
filename = op.join(temp_folder, "data.pkl")
joblib.dump(data, filename)
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 0cafae42ea2aa..979053d4a61e2 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -54,7 +54,8 @@
from ..model_selection import ShuffleSplit
from ..model_selection._validation import _safe_split
from ..metrics.pairwise import rbf_kernel, linear_kernel, pairwise_distances
-from ..utils.fixes import threadpool_info
+from ..utils.fixes import sp_version
+from ..utils.fixes import parse_version
from ..utils.validation import check_is_fitted
from . import shuffle
@@ -648,6 +649,8 @@ def _set_checking_parameters(estimator):
# avoid deprecated behaviour
params = estimator.get_params()
name = estimator.__class__.__name__
+ if name == "TSNE":
+ estimator.set_params(perplexity=2)
if "n_iter" in params and name != "TSNE":
estimator.set_params(n_iter=5)
if "max_iter" in params:
@@ -744,6 +747,11 @@ def _set_checking_parameters(estimator):
if name == "OneHotEncoder":
estimator.set_params(handle_unknown="ignore")
+ if name == "QuantileRegressor":
+ # Avoid warning due to Scipy deprecating interior-point solver
+ solver = "highs" if sp_version >= parse_version("1.6.0") else "interior-point"
+ estimator.set_params(solver=solver)
+
if name in CROSS_DECOMPOSITION:
estimator.set_params(n_components=1)
@@ -1419,6 +1427,10 @@ def check_fit2d_1sample(name, estimator_orig):
if name == "OPTICS":
estimator.set_params(min_samples=1)
+ # perplexity cannot be more than the number of samples for TSNE.
+ if name == "TSNE":
+ estimator.set_params(perplexity=0.5)
+
msgs = [
"1 sample",
"n_samples = 1",
@@ -2105,22 +2117,6 @@ def check_classifiers_one_label(name, classifier_orig):
assert_array_equal(classifier.predict(X_test), y, err_msg=error_string_predict)
-def _create_memmap_backed_data(numpy_arrays):
- # OpenBLAS is known to segfault with unaligned data on the Prescott architecture
- # See: https://github.com/scipy/scipy/issues/14886
- has_prescott_openblas = any(
- True
- for info in threadpool_info()
- if info["internal_api"] == "openblas"
- # Prudently assume Prescott might be the architecture if it is unknown.
- and info.get("architecture", "prescott").lower() == "prescott"
- )
- return [
- create_memmap_backed_data(array, aligned=has_prescott_openblas)
- for array in numpy_arrays
- ]
-
-
@ignore_warnings # Warnings are raised by decision function
def check_classifiers_train(
name, classifier_orig, readonly_memmap=False, X_dtype="float64"
@@ -2138,7 +2134,7 @@ def check_classifiers_train(
X_b -= X_b.min()
if readonly_memmap:
- X_m, y_m, X_b, y_b = _create_memmap_backed_data([X_m, y_m, X_b, y_b])
+ X_m, y_m, X_b, y_b = create_memmap_backed_data([X_m, y_m, X_b, y_b])
problems = [(X_b, y_b)]
tags = _safe_tags(classifier_orig)
@@ -2806,7 +2802,7 @@ def check_regressors_train(
y_ = y
if readonly_memmap:
- X, y, y_ = _create_memmap_backed_data([X, y, y_])
+ X, y, y_ = create_memmap_backed_data([X, y, y_])
if not hasattr(regressor, "alphas") and hasattr(regressor, "alpha"):
# linear regressors need to set alpha, but not generalized CV ones
diff --git a/sklearn/utils/extmath.py b/sklearn/utils/extmath.py
index 2521990e6cc68..089bd2efadff5 100644
--- a/sklearn/utils/extmath.py
+++ b/sklearn/utils/extmath.py
@@ -216,7 +216,7 @@ def randomized_range_finder(
# Generating normal random vectors with shape: (A.shape[1], size)
Q = random_state.normal(size=(A.shape[1], size))
- if A.dtype.kind == "f":
+ if hasattr(A, "dtype") and A.dtype.kind == "f":
# Ensure f32 is preserved as f32
Q = Q.astype(A.dtype, copy=False)
@@ -243,6 +243,7 @@ def randomized_range_finder(
# Sample the range of A using by linear projection of Q
# Extract an orthonormal basis
Q, _ = linalg.qr(safe_sparse_dot(A, Q), mode="economic")
+
return Q
diff --git a/sklearn/utils/fixes.py b/sklearn/utils/fixes.py
index b0074ae7e3a18..cdd63e00cd381 100644
--- a/sklearn/utils/fixes.py
+++ b/sklearn/utils/fixes.py
@@ -163,3 +163,10 @@ def threadpool_info():
threadpool_info.__doc__ = threadpoolctl.threadpool_info.__doc__
+
+
+# TODO: Remove when SciPy 1.9 is the minimum supported version
+def _mode(a, axis=0):
+ if sp_version >= parse_version("1.9.0"):
+ return scipy.stats.mode(a, axis=axis, keepdims=True)
+ return scipy.stats.mode(a, axis=axis)
diff --git a/sklearn/utils/metaestimators.py b/sklearn/utils/metaestimators.py
index 1cee8d1d42cf4..b0b20db7a5017 100644
--- a/sklearn/utils/metaestimators.py
+++ b/sklearn/utils/metaestimators.py
@@ -200,7 +200,7 @@ def __init__(self, fn, delegate_names, attribute_name):
def _check(self, obj):
warnings.warn(
"if_delegate_has_method was deprecated in version 1.1 and will be "
- "removed in version 1.3. Use if_available instead.",
+ "removed in version 1.3. Use available_if instead.",
FutureWarning,
)
diff --git a/sklearn/utils/multiclass.py b/sklearn/utils/multiclass.py
index 5311076e64eb8..c72846f9aa323 100644
--- a/sklearn/utils/multiclass.py
+++ b/sklearn/utils/multiclass.py
@@ -150,7 +150,7 @@ def is_multilabel(y):
warnings.simplefilter("error", np.VisibleDeprecationWarning)
try:
y = np.asarray(y)
- except np.VisibleDeprecationWarning:
+ except (np.VisibleDeprecationWarning, ValueError):
# dtype=object should be provided explicitly for ragged arrays,
# see NEP 34
y = np.array(y, dtype=object)
@@ -292,7 +292,7 @@ def type_of_target(y, input_name=""):
warnings.simplefilter("error", np.VisibleDeprecationWarning)
try:
y = np.asarray(y)
- except np.VisibleDeprecationWarning:
+ except (np.VisibleDeprecationWarning, ValueError):
# dtype=object should be provided explicitly for ragged arrays,
# see NEP 34
y = np.asarray(y, dtype=object)
diff --git a/sklearn/utils/sparsefuncs.py b/sklearn/utils/sparsefuncs.py
index d53741c044c47..64a86cf1180bf 100644
--- a/sklearn/utils/sparsefuncs.py
+++ b/sklearn/utils/sparsefuncs.py
@@ -385,7 +385,7 @@ def inplace_swap_row(X, m, n):
def inplace_swap_column(X, m, n):
"""
- Swaps two columns of a CSC/CSR matrix in-place.
+ Swap two columns of a CSC/CSR matrix in-place.
Parameters
----------
diff --git a/sklearn/utils/tests/test_estimator_html_repr.py b/sklearn/utils/tests/test_estimator_html_repr.py
index 91644819864eb..4624896abd307 100644
--- a/sklearn/utils/tests/test_estimator_html_repr.py
+++ b/sklearn/utils/tests/test_estimator_html_repr.py
@@ -309,3 +309,14 @@ def test_show_arrow_pipeline():
'class="sk-toggleable__label sk-toggleable__label-arrow">Pipeline'
in html_output
)
+
+
+def test_invalid_parameters_in_stacking():
+ """Invalidate stacking configuration uses default repr.
+
+ Non-regression test for #24009.
+ """
+ stacker = StackingClassifier(estimators=[])
+
+ html_output = estimator_html_repr(stacker)
+ assert html.escape(str(stacker)) in html_output
diff --git a/sklearn/utils/tests/test_extmath.py b/sklearn/utils/tests/test_extmath.py
index ece7c180300a1..a9ba7a96685d6 100644
--- a/sklearn/utils/tests/test_extmath.py
+++ b/sklearn/utils/tests/test_extmath.py
@@ -7,7 +7,6 @@
import numpy as np
from scipy import sparse
from scipy import linalg
-from scipy import stats
from scipy.sparse.linalg import eigsh
from scipy.special import expit
@@ -20,6 +19,7 @@
from sklearn.utils._testing import assert_array_equal
from sklearn.utils._testing import assert_array_almost_equal
from sklearn.utils._testing import skip_if_32bit
+from sklearn.utils.fixes import _mode
from sklearn.utils.extmath import density, _safe_accumulator_op
from sklearn.utils.extmath import randomized_svd, _randomized_eigsh
@@ -57,7 +57,7 @@ def test_uniform_weights():
weights = np.ones(x.shape)
for axis in (None, 0, 1):
- mode, score = stats.mode(x, axis)
+ mode, score = _mode(x, axis)
mode2, score2 = weighted_mode(x, weights, axis=axis)
assert_array_equal(mode, mode2)
diff --git a/sklearn/utils/tests/test_testing.py b/sklearn/utils/tests/test_testing.py
index fca7a07b14c19..75f35a3dea83c 100644
--- a/sklearn/utils/tests/test_testing.py
+++ b/sklearn/utils/tests/test_testing.py
@@ -702,16 +702,19 @@ def test_create_memmap_backed_data(monkeypatch, aligned):
assert registration_counter.nb_calls == 3
input_list = [input_array, input_array + 1, input_array + 2]
- if aligned:
- with pytest.raises(
- ValueError, match="If aligned=True, input must be a single numpy array."
- ):
- create_memmap_backed_data(input_list, aligned=True)
- else:
- mmap_data_list = create_memmap_backed_data(input_list, aligned=False)
- for input_array, data in zip(input_list, mmap_data_list):
- check_memmap(input_array, data)
- assert registration_counter.nb_calls == 4
+ mmap_data_list = create_memmap_backed_data(input_list, aligned=aligned)
+ for input_array, data in zip(input_list, mmap_data_list):
+ check_memmap(input_array, data)
+ assert registration_counter.nb_calls == 4
+
+ with pytest.raises(
+ ValueError,
+ match=(
+ "When creating aligned memmap-backed arrays, input must be a single array"
+ " or a sequence of arrays"
+ ),
+ ):
+ create_memmap_backed_data([input_array, "not-an-array"], aligned=True)
@pytest.mark.parametrize("dtype", [np.float32, np.float64, np.int32, np.int64])
diff --git a/sklearn/utils/tests/test_utils.py b/sklearn/utils/tests/test_utils.py
index 82be82afa5eed..4fc483e5a85b2 100644
--- a/sklearn/utils/tests/test_utils.py
+++ b/sklearn/utils/tests/test_utils.py
@@ -472,8 +472,13 @@ def test_safe_indexing_pandas_no_settingwithcopy_warning():
X = pd.DataFrame({"a": [1, 2, 3], "b": [3, 4, 5]})
subset = _safe_indexing(X, [0, 1], axis=0)
+ if hasattr(pd.errors, "SettingWithCopyWarning"):
+ SettingWithCopyWarning = pd.errors.SettingWithCopyWarning
+ else:
+ # backward compatibility for pandas < 1.5
+ SettingWithCopyWarning = pd.core.common.SettingWithCopyWarning
with warnings.catch_warnings():
- warnings.simplefilter("error", pd.core.common.SettingWithCopyWarning)
+ warnings.simplefilter("error", SettingWithCopyWarning)
subset.iloc[0, 0] = 10
# The original dataframe is unaffected by the assignment on the subset:
assert X.iloc[0, 0] == 1
diff --git a/sklearn/utils/tests/test_validation.py b/sklearn/utils/tests/test_validation.py
index a8078a2b39416..478617bcc4031 100644
--- a/sklearn/utils/tests/test_validation.py
+++ b/sklearn/utils/tests/test_validation.py
@@ -358,21 +358,13 @@ def test_check_array():
Xs = [X_csc, X_coo, X_dok, X_int, X_float]
accept_sparses = [["csr", "coo"], ["coo", "dok"]]
- for X, dtype, accept_sparse, copy in product(Xs, dtypes, accept_sparses, copys):
- with warnings.catch_warnings(record=True) as w:
- X_checked = check_array(
- X, dtype=dtype, accept_sparse=accept_sparse, copy=copy
- )
- if (dtype is object or sp.isspmatrix_dok(X)) and len(w):
- # XXX unreached code as of v0.22
- message = str(w[0].message)
- messages = [
- "object dtype is not supported by sparse matrices",
- "Can't check dok sparse matrix for nan or inf.",
- ]
- assert message in messages
- else:
- assert len(w) == 0
+ # scipy sparse matrices do not support the object dtype so
+ # this dtype is skipped in this loop
+ non_object_dtypes = [dt for dt in dtypes if dt is not object]
+ for X, dtype, accept_sparse, copy in product(
+ Xs, non_object_dtypes, accept_sparses, copys
+ ):
+ X_checked = check_array(X, dtype=dtype, accept_sparse=accept_sparse, copy=copy)
if dtype is not None:
assert X_checked.dtype == dtype
else:
@@ -1286,15 +1278,20 @@ def test_check_psd_eigenvalues_valid(
if not enable_warnings:
w_type = None
- w_msg = ""
- with pytest.warns(w_type, match=w_msg) as w:
- assert_array_equal(
- _check_psd_eigenvalues(lambdas, enable_warnings=enable_warnings),
- expected_lambdas,
- )
if w_type is None:
- assert not w
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", PositiveSpectrumWarning)
+ lambdas_fixed = _check_psd_eigenvalues(
+ lambdas, enable_warnings=enable_warnings
+ )
+ else:
+ with pytest.warns(w_type, match=w_msg):
+ lambdas_fixed = _check_psd_eigenvalues(
+ lambdas, enable_warnings=enable_warnings
+ )
+
+ assert_allclose(expected_lambdas, lambdas_fixed)
_psd_cases_invalid = {
+
+
+
+---
+
+## Reference
+- [color-hex](https://www.color-hex.com): Glossary of Color Palettes
+
+## Other
+You can find more variations of the logos here: https://github.com/scikit-learn/blog/tree/main/assets/images
diff --git a/doc/logos/brand_colors/colorswatch_29ABE2_cyan.png b/doc/logos/brand_colors/colorswatch_29ABE2_cyan.png
new file mode 100644
index 0000000000000..b014a859dd4b9
Binary files /dev/null and b/doc/logos/brand_colors/colorswatch_29ABE2_cyan.png differ
diff --git a/doc/logos/brand_colors/colorswatch_9B4600_brown.png b/doc/logos/brand_colors/colorswatch_9B4600_brown.png
new file mode 100644
index 0000000000000..379400786ef56
Binary files /dev/null and b/doc/logos/brand_colors/colorswatch_9B4600_brown.png differ
diff --git a/doc/logos/brand_colors/colorswatch_F7931E_orange.png b/doc/logos/brand_colors/colorswatch_F7931E_orange.png
new file mode 100644
index 0000000000000..5b22b575ac411
Binary files /dev/null and b/doc/logos/brand_colors/colorswatch_F7931E_orange.png differ
diff --git a/doc/logos/scikit-learn-logo.svg b/doc/logos/scikit-learn-logo.svg
index 523a656943772..362542602e0ae 100644
--- a/doc/logos/scikit-learn-logo.svg
+++ b/doc/logos/scikit-learn-logo.svg
@@ -1,110 +1 @@
-
-
-
-
\ No newline at end of file
+
diff --git a/doc/make.bat b/doc/make.bat
index fa8e7171ea7e6..b7e269a6a7836 100644
--- a/doc/make.bat
+++ b/doc/make.bat
@@ -9,7 +9,7 @@ if NOT "%PAPER%" == "" (
set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
)
-if "%1" == "" goto help
+if "%1" == "" goto html-noplot
if "%1" == "help" (
:help
@@ -42,6 +42,7 @@ if "%1" == "html" (
)
if "%1" == "html-noplot" (
+ :html-noplot
%SPHINXBUILD% -D plot_gallery=0 -b html %ALLSPHINXOPTS% %BUILDDIR%/html
echo.
echo.Build finished. The HTML pages are in %BUILDDIR%/html
diff --git a/doc/modules/clustering.rst b/doc/modules/clustering.rst
index 881f884453fc1..13b2004a41223 100644
--- a/doc/modules/clustering.rst
+++ b/doc/modules/clustering.rst
@@ -141,7 +141,7 @@ K-means
The :class:`KMeans` algorithm clusters data by trying to separate samples in n
groups of equal variance, minimizing a criterion known as the *inertia* or
within-cluster sum-of-squares (see below). This algorithm requires the number
-of clusters to be specified. It scales well to large number of samples and has
+of clusters to be specified. It scales well to large numbers of samples and has
been used across a large range of application areas in many different fields.
The k-means algorithm divides a set of :math:`N` samples :math:`X` into
diff --git a/doc/modules/compose.rst b/doc/modules/compose.rst
index 2a2b007783f27..c797eb288c6e5 100644
--- a/doc/modules/compose.rst
+++ b/doc/modules/compose.rst
@@ -202,7 +202,7 @@ each configuration.
The parameter ``memory`` is needed in order to cache the transformers.
``memory`` can be either a string containing the directory where to cache the
-transformers or a `joblib.Memory