The issue probably comes from the fact that the serialization uses %f for the values which truncates to 6 places by default whatever the dtype of the input. For np.float64 we should probably use something like %0.16f instead. WDTY?

We should check that libsvm can accept such long values as input format though.

I've yet to read this through, but wouldn't it be easier to add a single parameter indicating the number of lines to read, then pass the same file-like to load_svmlight_file multiple times?

(Also, how large are your matrices? I was under the impression that you can never store more than 2**31 elements, or can you store actually that number of rows and columns?)

I've yet to read this through, but wouldn't it be easier to add a single parameter indicating the number of lines to read, then pass the same file-like to load_svmlight_file multiple times?

I would like to be able os.stat a big svmlight file (several GB) and divide it into mt_size / n_chunks that would fit in memory to be parsed by n_workers in // without having to do a first sequential scan to count the lines and find the bytes offsets. Workers would then dump intermediate CSR data struct on the filesystem and a second pass would just aggregate them all into a single memmapped CSR with n_features being the max of the observed n_features on the indivual chunks.

(Also, how large are your matrices? I was under the impression that you can never store more than 2**31 elements, or can you store actually that number of rows and columns?)

I would like to be able to deal with matrices of the scale of the PASCAL large scale challenge:

$ time wc ~/Desktop/alpha/alpha.txt 
  500000 250500000 3590403790 /Users/oliviergrisel/Desktop/alpha/alpha.txt

real    0m27.502s
user    0m24.754s
sys 0m2.207s

This indeed won't fit on a single CSR:

>>> 3590403790. / 2 ** 31
1.6719120508059859

That's unfortunate but I guess I can split it into 10 CSR chunks or so and then use the partial_fit method of the SGDClassifier class (for instance) to deal with this limitation.

Would be great to have support for np.int64 indices in scipy.sparse though...

Splitting lines is CPU bound (at least wc -l is). Hence being able to seek forward several workers on a multicore machine should bring a speed up.

Actually, wc -l is IO bound on my laptop (with SSD) but wc (parsing more similar to our svmlight parser) is CPU bound.

Interesting / useful contrib!

It could be useful to have a way to estimate the actual size once a file chunk has been converted to CSR format.

Minor remark: for me, it would be more natural to use offset and length parameters rather than offset_min and offset_max.

Then length would be equivalent to offset_max - offset_min? Why not. I might do the change later today.

Ok I rebased, added more tests and fixed a bug. I also switched to @mblondel suggested API (offset + length). I think this is ready for merge to master. WDYT?

I think it would be useful to add a generator function, say load_svmlight_file_chunks, that takes a n_chunks parameter and produces (X, y) pairs.

You assume that n_features is user-given, right? You might want to raise an exception if that's not the case.

I think it would be useful to add a generator function, say load_svmlight_file_chunks, that takes a n_chunks parameter and produces (X, y) pairs.

I was thinking to write a parallel, out-of-core conversion tool that would generate joblib dumps of chunked data in a folder instead. Do you think both would be useful?

You assume that n_features is user-given, right? You might want to raise an exception if that's not the case.

For my conversion tool I want to do a single parsing pass over the data and record the n_features detected in each chunk, take the max and re-save the datasets iteratively by padding in non-parsing hence fast second pass over the previously extracted chunks.

Inferring n_features seems a bit expensive.

We could reproject the data while it is loaded from the svmlight file using a FeatureHasher. This way, n_features can be safely fixed.

Another thing I would like to check is whether a crude upper bound on n_features would work. The training time of solvers like SGDClassifier or LinearSVC with dual=True is not affected by the number of features (the training time of CD-based solvers is).

My goal is to convert the svmlight file into a contiguous datastructure save on the hard drive once a never have to parse the svmlight file again. It's a dataset format conversion tool. I don't want to do learning on the fly in my case.

I simply use joblib for these purposes :)

Yes but in this case I need to do it out of core as the svmlight file is 11GB and the dense representation is twice as big (without compression) and I want to detect the number of features on the fly so I need to do a second non-parsing pass to pad the previously extracted arrays with zero features.

The svmlight format is very useful for preparing the data in one programming language and learning the model in another. It's really easy to write a script for outputting data to this format.

I think the chunk loading support can be merged as it is. It's already useful for advanced users. I am not sure we want to implement a generic out-of-core converter in the library. Maybe it would be better implemented as an example in a benchmark script based on the mnist8m dataset. I will do that in another PR later.

Could you rebase?

@jnothman thanks for the review. I added a test to check for all the possible of the byte offset of small dataset (along with query ids). The exhaustive test run in 500ms. This should cover all the boundary cases you mentioned.

@jnothman I fixed (workaround) the broken tests with old versions of scipy.

Mind adding a what's new before merge, @ogrisel?

@jnothman done. I also rebased on top of current master to insert the entry at the right location. Let's see if CI is still green.

Merged! The scipy version check in the test was to lax. I updated it.

I'm tempted to say: Thanks for your patience, @ogrisel ;)

It's pretty exciting to close an issue #<1000