I just ran into this.

from sklearn.datasets import fetch_openml
mnist = fetch_openml("mnist_784")

takes 8 seconds on my machine.
I would suggest not waiting for OpenML.

Sorry, last try was 40s actually. Yes, master.
Maybe the chunking made it slower, not sure?

I get around 16 seconds on master, and 20 seconds on 0.20.2.

Regarding the arff mnist dataset, the biggest bottlenecks are the _arff._decode_values and _arff._parse_values which together uses about 80% of the time. Specifically, _arff._decode_values calls float 70k times and _arff._parse_values runs a regex 70k times.

For mnist, the arff dataset uses 15.5 MB. If we were to use np.savez_compressed to cache the mnist dataset will use 23MB.

If we save the numpy array, it would be good to remove the data from the arff file and let it keep the metadata.

Thanks for checking. How is the numpy file bigger than the arff file? That makes no sense to me.

How much faster is it to cache?
We could also use joblib memory for caching, which is I think what I originally implemented.

It's very possible that we can speed up liac-arff still, even though I tried hard to get it as fast as possible without moving to C.

When comparing its performance to pandas.read_csv, if we use Cython/C/etc there is certainly room for improvement. The question is whether that additional effort would be justified. A mid-term solution to use some better file format, which could also solve other issues with arff.

@amueller

Thanks for checking. How is the numpy file bigger than the arff file? That makes no sense to me.

I suspect that the numpy are using more space stores the data as float64. The original ARFF file contains a bunch of 0s and a few uint8s to store the images.

How much faster is it to cache?
The caching will be slower, because of the following:

1. Download arff file and save arff to disk.
2. Read ARFF file and check if we support caching it as a numpy array or a scipy sparse matrix.
3. Write numpy array into cache. (For mnist this takes ~8 seconds)
4. Maybe remove the data from the arff file, leaving the metadata.

Loading the numpy array takes ~2 seconds for the mnist dataset.

We could also use joblib memory for caching, which is I think what I originally implemented.
joblib is not currently used.

For saving sparse matrices, we can use scipy.sparse.save_npz.

Related issue: renatopp/liac-arff#75

What is preventing us from using scipy.io.arff.loadarff?

Our options are to download arff and convert locally once, or to download a binary format, right?
If we download a binary format, we need to be able to read it, though, which requires a dependency. I'm sure openml would be happy to add an additional endpoint for us to get parquet, but then we need to depend on a parquet loader.

In a completely unscientific experiment, scipy took 27s to load the mnist arff and liac-arff took 17s.
In a rerun scipy took 22 and liac-arff again 17.

@raghavrv's pyarff: https://github.com/pyarff/pyarff
takes 0.8 seconds.

Maybe that would be worth taking up for dense datasets?

before I delegate @thomasjpfan to another thing he should probably finish up all his amazing half-finished PRs so if someone else wants to work on this in the meantime ;)