I noticed that this fetcher is extremely memory inefficient and slow. For all practical purposes I think it's much better to use fetch_openml

Which is also slow (ARFF parsing in pure python) but I imagine less so :)

Just a side note: when reviewing #10027 I noticed that this fetcher is extremely memory inefficient and slow. For all practical purposes I think it's much better to use fetch_openml.

fetch_openml is more memory efficient because it does parsing the arff file in chunks.

Which is also slow (ARFF parsing in pure python) but I imagine less so :)

The (near?) future, we can use Parquet files! A parquet parser would be harder to vendor, so it may lead us down a path of using optional dependencies. (Like how pandas.read_parquet works)

XREF: openml/openml-python#1032
XREF: renatopp/liac-arff#120
XREF: #11821

The (near?) future, we can use Parquet files! A parquet parser would be harder to vendor, so it may lead us down a path of using optional dependencies.

Great! Looking forward to that.

Really good news :)

Which is also slow (ARFF parsing in pure python) but I imagine less so :)

I cannot even use the scikit-learn loader without getting my 16GB RAM system to start swapping while fetch_openml uses less than 1GB and completes in a few seconds:

https://github.com/scikit-learn/scikit-learn/pull/10027/files#r595370382