The data attribute will contain the whole data frame. Usually a pandas dataframe is use to explore the data, using groupbys, etc. Having the target in the dataframe helps with this aspect.

What about if return_X_y=True?

I agree that including the target in the frame corresponds to common pandas usage, but it doesn't correspond to Scikit-learn convention. I'd rather leave the target out unless you have a more compelling reason otherwise.

I agree that splitting up the dataframe into target and data matches the Scikit-learn convention more. If we break up data and target, we would need to guide users to recombine the dataset to do any exploratory data analysis that needs the target in the dataframe. This is a little counter to what is expected when loading a dataframe from a csv, where the target is included. Users are use to breaking up the dataframe themselves.

In the end, I am okay with going either way on this matter.

This PR has been updated with:

1. An nrows parameter in fetch_openml to control how many rows to consume at a time when constructing the dataframe.
2. There is a less copying in _convert_arff_data_dataframe.
3. Features where openml says the data is is_ignore or is_row_identifier are remove from the dataframe. If a dataset contains these columns, they are removed, and the dataframe is copied. This forces fetch_openml to return a non-view dataframe.
4. Raises an error when return_X_y=True and return_frame=True. To correctly follow scikit-learn convention, X could still be a dataframe, but y can be reshaped into a 1-D array (if there is one target). Letting y be a numpy array appears to contradict return_frame=True.

This PR was updated with an additional bunch entry, dataframe, which still includes the target. When return_frame=True, both data and target are None. A user would need to use target_columns and feature_columns to split features from the target. Adding a dataframe entry helps distinguish this API from the rest of the sklearn.datasets api.

I think the common use case for using a dataframe is to do EDA (using the target), and then use the target_columns and feature_columns to split the data. The user would also need to be careful with the shape of y, which is already the case when working with dataframes.

Lets say we split the dataframe up into data and target, the target would need to be reshaped, lose its pandas categorical dtype information, and its column name. If we want a user to reconstruct the original dataframe, they would need to manually reconstruct the categorical dtype and place the original column name back. All of this work would be required to do EDA.

Currently the sklearn.dataset api is designed to return a data and target that can be directly fed into a scikit-learn estimator. I am envisioning a new dataframe entry, where a user would want to explore the data first, before building a model.

(although even there the user should probably not look at the target - and
arguably the features - on their test set)

There are some plots that would like having the target in the dataframe. For example, the iris dataset with seaborn, uses the target, species, to control the hue.

We are forcing users to learn a new way of
interacting with datasets (whether changing to fetch_openml from load_iris,
or vice versa) despite them being functionally required to use return_frame
in many cases. Does that make sense?

This makes sense. Keeping the datasets API consistent is important. Okay, I will update this PR to split the features and the target.

I should also point out that return_frame='auto' could be a nice feature,
for datasets with any non-numeric features. The current approach precludes
that possibility.

This sounds reasonable. It does not feel like too much magic.

This PR was updated with:

1. Renames data_column to data_columns in the implementation.
2. Uses get_chunk_n_rows to get chunksize.
3. Only create Bunch object in one place.
4. Filters out columns as the dataframe is being read.

PR updated with:

1. Test to compare as_frame=True and as_frame=False on a purely numerical dataset.

@thomasjpfan Thanks. Maybe we can have some example that could benefit from it now.

I'm really excited that this was added! Thanks to everyone who worked on it!

@thomasjpfan thanks for taking this to the finish line !