(The PyPy test failure should be the same as in the empty/intp sequence parsing PR: PyPy had a bug with formatting %(100)R, it is fixed, and for now we should just ignore it.)

I admit I am not a big fan of converting working python code into 90% working C code by hand. It seems like the wrong approach for 2021.

I would be interested to see what happens if we use cython to convert loadtxt (and its functions) to C. Does it achieve a similar speedup?

In any case, can we leave this as an external package for now, especially given the open issues and design decisions. Perhaps this PR could target nreadtext instead of NumPy?

I will probably write more on the mailing list later. The quick answer is, that I doubt it will help much, but can try quickly. I currently see two alternatives, but maybe I am lacking vision:

• Use Python's csv module with fromiter
• Try cython to convert the exist code largely +as is.

I doubt either will get within a factor of 2 for many use-cases, but can experiment. Moving the existing code to Cython does not seem a likely an improvement to me (since this is little Python interfacing code).

Assuming I am not surprised by the above tries, we have to decide how important it is to us to get:

• A parser that is not at least 2× slower than pandas (for many use-cases)
• access to CSV features (e.g. quoting)

For something that is, for better or worse, probably core functionality. As much as I dislike the amount of code and would love to off-load or even ignore it, the status-quo where we write in our tutorial "please use pandas" is not great. (Personally, I always felt that if loading a CSV is too slow, it just proofs that I should not use CSV for that task. But in practice, I have to admit, I doubt that is the reality out in the wild.)

Fast reading of data in huge text files seems to be important, at least comparisons are popular :) Julia boasts it is faster than Pandas, Dask touts the advantages of parallelization, so on and so forth. I don't deal with huge text files, but use of low level languages to speed up common use cases is pretty much why NumPy and SciPy exist, so I am not opposed in principle. The only problem I see is trading off the complexity/maintainability of a low level language vs Python. But I also note that NumPy is becoming more complex with SIMD and C++ templates, it is perhaps in the nature of things that performance drives development.

3k lines for a 50% improvement over pandas when using maximum float precision seems like a less-than-ideal ratio (and slower if one is happy with the default).

>>> c = pd.read_csv("test.csv", delimiter=",")
96.2 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
>>> d = pd.read_csv("test.csv", delimiter=",", float_precision="round_trip")
322 ms ± 7.27 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
>>> e = _loadtxt("test.csv", delimiter=",")
215 ms ± 2.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The pandas read_csv has a really large user base and so benefits accordingly in terms of edge cases.

I would be interested to see what happens if we use cython to convert loadtxt (and its functions) to C. Does it achieve a similar speedup?

IME one big advantage of Cython over C -- especially if the Cython is pythonic and uses cython cimports rather than C function calls -- is the breadth of the base that is comfortable making PRs.

Looking at this post it seems to me that unless one wants to do a multithreaded reader then the best case gains are pretty limited (if one wants to compare to Julia as a reference point).

3k lines for a 50% improvement over pandas when using maximum float precision seems like a less-than-ideal ratio (and slower if one is happy with the default).

Yes, IIRC, compared to pandas this has basically parity or up to 50% faster (maybe more for integer parsing in some cases). But, I think you will find the C code to be much more maintainable and much easier to extend. Some notes:

• To get feature-parity with loadtxt, the pandas version must use its Python backend which is also extremely complex (and very slow of course). This is for things like:

  • Reading any file that isn't a raw utf-8 file.
  • Use any control character that is not ASCII. E.g. splitting by whitespace delimiter=None doesn't work, pandas only supports ascii whitespace with any decent speed (even with delimiter=',' non-ascii whitespace will not get stripped when ascii-whitespace does get stripped.

  On the contrary, here the reading itself is still largely using Python API to keep it simple and flexible (with a line-by-line vs. chunked mode adding a bit of complexity for ~30% speedup, I admit). Meaning that all of those cases are covered by the same code.

• I am very sure that if you will have much more problems extending the pandas code. I wrote this in a way that it should be straight forward to e.g. allow users to supply parsers written in C directly, or allow swapping out our parsers (if you want to have that fast float parser). I like the idea to make it so that users can supply converters=faster_float_parser_written_in_C. (yes, we would have to nail down public API for that, but the general design is there.)

• The pandas code is much less maintainable also IMO. It has as many lines of code, but is not nearly as well structured.

Is Cython better? I guess opinions will differ on that. It is probably more accessible to contributions, but honestly, I am not sure I like Cython for code that is "heavy C". It is great for the interface code, but the interface code isn't the big chunk here.

3k lines for a 50% improvement over pandas when using maximum float precision seems like a less-than-ideal ratio

Pandas has the reputation of being the best/fastest Python text reader, and a long term goal has been to make NumPy as good as Pandas. Reaching that goal probably requires doing most of the work in C. It would be interesting to see just what makes the difference when the code is C rather than Python -- reading, parsing, writing, etc. -- but it does seem clear that all the fastest readers are written in low level languages for the common case. Both R and Pandas do that, and Julia is probably in the low level language camp.

What are the benchmarks for the code? Is there something that exercises the non-float dtypes?

What are the benchmarks for the code? Is there something that exercises the non-float dtypes?

The benchmarks @seberg added in the OP (click on the little black triangle to open the details tab) were computed using bench_io from the benchmark suite. You should be able to test/reproduce locally w/ something like python runtests.py --bench-compare main add-npreadtext bench_io

My 2 euro cents:

• We have a ton of open loadtxt/genfromtxt issues, and this has been a mess for years. That plus the slowness justifies a rewrite.
• I wouldn't have had a very clear preference between adopting the Pandas reader (well-tested by now) or rewriting from scratch (cleaner, bit faster code) without spending quite a bit of time looking into the problem. I'd lean towards the Pandas one mostly to avoid spending a lot of developer time. That said, now that the work is mostly done I'm happy to trust @seberg & co that this PR is the right choice by now.
• It looks like a low-stakes decision to merge this PR once it's complete. It's then faster and better structured than the Pandas reader, and benchmarks look great. So if the tests pass (and ideally it fixes a bunch of bugs as well), then +1 for merging. In the (quite unlikely) case it turns out to be horribly buggy, revert it in 1.23.1 and try again or reconsider.

Is Cython better? I guess opinions will differ on that. It is probably more accessible to contributions, but honestly, I am not sure I like Cython for code that is "heavy C". It is great for the interface code, but the interface code isn't the big chunk here.

I tend to agree with this point of view. In addition, build system wise C is much nicer.

The test suite changes look fairly modest, and include some improved error messages it seems.

Yeah, the main changes I am aware of are:

• more restrictive parsing: no floats when storing integers, no underscore 100_000 support (can be gotten back using converters=float although a bit annoying if you try to use both versions, since the old one requires listing all columns.
• There is some strange stuff about counting lines vs. counting rows: loadtxt strange handling empty lines with max_rows #19785 (this can be fixed in any way pretty much, but needs a decision). Also if you pass in rows that contain newlines, things get a bit weird. I did not care much, because it seems super strange, but it is a change

The other things should really be improvements, like the better errors (although sometimes changed) – could still add error chaining there, especially for user provided converters.

In any case, thanks for taking a look! I will be off mostly for the next 1.5 weeks, so probably no updates :).

I can't resist to comment here, so sorry in advance ;)

Long time ago I have written my own CSV parser. It had support for quotes, I think that should be pretty much standard. CSV parsers are just inherently complex and need to handle legacy, since this is what CSV is about to a large amount ("I want to read that old file from this legacy system XYZ here"). I totally agree with @seberg that it is about the structure of the code rather than the language it is written in (C or Python) to keep this complexity as low as possible and a CSV parser maintainable. Maintainability and reliability should be more important than speed anyway, since there is not so much speed to be gained, as even GPUs can only accelerate by a factor of 2-3x, compare

https://medium.com/rapids-ai/reading-larger-than-memory-csvs-with-rapids-and-dask-e6e27dfa6c0f

Having CSV in a workflow is frustrating. It is always slow and when dealing with floats (so probably most of the time), it is a reason for precision loss. In my workflows, I typically convert CSV to HDF or .npy first and create intermediate, binary cache files if needed to have short cycle times ("save .py, automated run (e.g. with entr), see results" should take only a few seconds). However, one nice thing about CSV is that it can be easily appended to, just by writing to the end of the file and at this point, I'd like to advertise my pull request a little more:

#20321

This would make .npy appendable in the same way CSV can be appended to: just write some more data to the end of the file, no need to update headers.

And I'd also like to add that most CSV parsers I've encountered (C, C++, JavaScript, Java) are simply broken and unmaintained. The one from Pandas is indeed the best I've seen so far, probably because of its large user base.

We have a ton of open loadtxt/genfromtxt issues

It might be nice to see which of the 60 open issues with loadtxt or genfromtxt are relevant

Frankly, I don't think it closes a lot. It closes some performance (speed/memory) relevant ones (although some of them mention "genfromtxt" as well):

• numpy.loadtxt with skiprows much slower than custom generator #7480
• genfromtxt and loadtxt fails with memory error with big csv #8523 (for loadtxt)

It fixes/changes gh-19785 (as documented and we wanted to clean this part up slightly more), but I can also undo that. It also implements gh-9291, but it is just a small carrot if anything. A major thing is quoting support as e.g. requested in gh-2211 (but not for genfromtxt for now obviously).
I think there are a couple of others that may just not be listed anywhere, like:

• Using float for int conversions is clearly incorrect for some values.
• Better error messages

So does it fix a huge amount of issues? No, not really. It addresses the few big ones that meant that everyone is told not to use loadtxt: performance/memory use and CSV features like quoting – full features can now be achieved, such as escaping.
For most issues, moving it to C doesn't really change all that much about how hard/easy it would be to address them.

Warren had added support for things like decimal=",", I ripped it out because that chunk relied on vendoring 3000 lines of float parsing code and I prefer to start minimal. But that can be re-added also without vendoring, by adding it to a translation step (which really should not make a big difference on the scale of float parsing).

EDIT: There are a couple of rewordings and a few simple tests to be added. The only bigger thing is to review documentation. (There is one possibility that specifying things like quotechar="#", delimiter="#" can misbehave in some cases and should error explicit, although I somewhat doubt it.)

The last of the small fixups that were queued up from previous reviews have been addressed, so this one is ready for review!

something went wrong with CI

Thanks Matti! A typo (or two) snuck into the last docs update; fixed now. the remaining failure is due to the scipy docs (general doc CI failure currently).

There will be some follow-on PRs but this looks good so far.

Yes, the main potential follow-ups will be looking into:

• removing that Python 2 compatibility encoding="bytes" shim.
• renaming skip_rows to skip_lines to make it clearer.

(Besides any potential issue fixes of course. I will probably send a brief heads-up email to the mailing list once it lands.)

Thanks @seberg, @rossbar, @WarrenWeckesser