✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 226e1f7. Link to the linter CI: here}

Hi @jeremiedbb

I think this can be reproduced without the ProcessPoolExecutor part. The code in the details produces this output:

Traceback (most recent call last):
  File "/Users/omar.salman/PycharmProjects/pythonProject/pandas_error.py", line 37, in main
    regressor.fit(X_train, y_train)
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/base.py", line 1473, in wrapper
    return fit_method(estimator, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/linear_model/_base.py", line 609, in fit
    X, y = self._validate_data(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/base.py", line 650, in _validate_data
    X, y = check_X_y(X, y, **check_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/utils/validation.py", line 1289, in check_X_y
    y = _check_y(y, multi_output=multi_output, y_numeric=y_numeric, estimator=estimator)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/utils/validation.py", line 1299, in _check_y
    y = check_array(
        ^^^^^^^^^^^^
  File "/Users/omar.salman/.pyenv/versions/test-env/lib/python3.11/site-packages/sklearn/utils/validation.py", line 1107, in check_array
    array.flags.writeable = True
    ^^^^^^^^^^^^^^^^^^^^^
ValueError: cannot set WRITEABLE flag to True of this array

from multiprocessing.managers import BaseManager
import traceback

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression


class MemoryDataset:
    def __init__(self):
        self._ds = None

    def save(self, ds):
        self._ds = ds

    def load(self):
        return self._ds


class MyManager(BaseManager):
    pass


def main():
    MyManager.register("MemoryDataset", MemoryDataset, exposed=("save", "load"))
    rng = np.random.default_rng()
    n_samples = 1000
    X_train = pd.DataFrame(rng.random((n_samples, 4)), columns=list('ABCD'))
    y_train = pd.Series(rng.random(n_samples))
    manager = MyManager()
    manager.start()
    dataset = manager.MemoryDataset()
    dataset.save((X_train, y_train))
    regressor = LinearRegression()
    X_train, y_train = dataset.load()
    try:
        regressor.fit(X_train, y_train)
    except Exception as _:
        print(traceback.format_exc())
    return regressor


if __name__ == '__main__':
    main()

Could this have something to do with pandas.Series objects being loaded from a read only buffer which is why we can't set the flags? Consider the following code which can also reproduce the above exception

import traceback

import joblib
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression


def main():
    rng = np.random.default_rng()
    n_samples = 1000
    X_train = pd.DataFrame(rng.random((n_samples, 4)), columns=list('ABCD'))
    y_train_df = pd.Series(rng.random(n_samples))
    y_filename = "y_train.pkl"
    joblib.dump(y_train_df, y_filename)
    y_train = joblib.load(y_filename, mmap_mode="r")
    regressor = LinearRegression()
    try:
        regressor.fit(X_train, y_train)
    except Exception as _:
        print(traceback.format_exc())
    return regressor


if __name__ == '__main__':
    main()

@OmarManzoor it's more subtle. In your reproducer, the Series is backed by a read-only buffer, so the array is read-only and we try to make it writeable which is not possible. In the original reproducer, the array is already writeable, and just the attempt to szet the writeable flag to the same value it already has (i.e. True) fails.

For the more general issue regarding arrays/dataframes/series backed by read-only buffers, I opened a different PR, #29018.

@OmarManzoor it's more subtle. In your reproducer, the Series is backed by a read-only buffer, so the array is read-only and we try to make it writeable which is not possible. In the original reproducer, the array is already writeable, and just the attempt to szet the writeable flag to the same value it already has (i.e. True) fails.

Isn't it possible that the array in the original reproducer also comes from a read only buffer because of the memory manager and parallel processes involved.

I looked at the array flags just before calling

@jeremiedbb

The writeable flag is also set to True in the code snippet that I shared above. Here is the output of the debugger on the line before the error
occurs

if _is_pandas_df_or_series(array_orig) and hasattr(array, "flags"):

hasattr(array, "flags")
Out[2]: True
array.flags
Out[3]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False

@OmarManzoor, Are you sure it's not the check_array of X (this one is not backed by a read-only buffer) ? because when I do the same thing, I get

  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False

for the call of check_array on y

Are you sure it's not the check_array of X

Yes I made the mistake of checking the first call which is for X. For y its False as you stated.

Given that #29018 fixes more than what this PR does and that I know think that we should include it in 1.5.1, let's leave this one for now. I'm leaving it open in the mean time in case we finally don't want to include #29018 in 1.5.1 but let's move the reviews and discussions there.