We often get issues related to memory consumption and don't deal with them particularly well. Indeed, Scikit-learn should be at home on commodity hardware like developer/researcher laptops.

Some operations can be performed chunked, so that the result is computed in constant (or O(n)) memory relative to some current O(n) (O(n^2)) consumption. Examples include: getting the argmin and min of all pairwise distances (currently done with an ad-hoc parameter to pairwise_distances_argmin_min), calculating silhouette score (#1976), getting nearest neighbors with brute force (#7287), calculating standard deviation of every feature (#5651).

It's not very helpful to provide this "how much constant memory" parameter in each function (because they're often called within nested code), so this PR instead makes a global config parameter of it. The optimisation is then transparent to the user, but still configurable.

At @rth's request, this PR has been cut back. The proposed changes to silhouette and neighbors can be seen here.

This PR (building upon my work with @dalmia) will therefore:

• add set_config(working_memory=n_mib)
• add pairwise_distances_chunked
• make use of the latter in nearest neighbors, silhouette and pairwise_distances_argmin_min
• deprecate batch_size in pairwise_distances_argmin_min

and thus:

• help towards fixing System freeze on Silhouette scoring #7175, MemoryError from sklearn.metrics.silhouette_samples #10279, [MRG] Block-wise silhouette calculation to avoid memory consumption #7177 (silhouette)
• Resolve [MRG] ENH: Added block_size parameter for lesser memory consumption #7979
• help towards fixing Batch pairwise_distances in neighbors to reduce memory consumption #7287 (neighbors)
• perhaps help towards Euclidean pairwise_distances slower for n_jobs > 1 #8216
• provide an interface for fixing Is it possible to reduce StandardScaler.fit() memory consumption? #5651
• it looks like there was some suggestion of using this kind of chunking in LMNN (FEAT Large Margin Nearest Neighbor implementation #8602)

TODO:

• attract other core devs' attention which I feel has been lacking despite the repeated popular interest in these issues!
• review my comments to @dalmia at [MRG] ENH: Added block_size parameter for lesser memory consumption #7979
• add tests for get_chunk_n_rows
• add tests for _check_chunk_size and any others for pairwise_distances_chunked
• add config documentation
• fix, perhaps, Is it possible to reduce StandardScaler.fit() memory consumption? #5651 with a chunked standard deviation implementation (can be a separate PR)
• perhaps review uses of gen_batches to see if they can use this parameter to be more principled in the choice of batch size: in most cases the batch size affects the model, so we can't just change it. And in other cases, it may make little difference. We could change mean_shift's use from fixed batch_size=500 to something sensitive to working_memory if we wish.
• add what's new entry
• add example to pairwise_distances_chunked that uses start and a tuple return
• benchmarking