The basic idea is not to iterate over all 127 ASCII characters and count their frequency on each line in _guess_delimiter but only over present characters, and just backfill zeros.

Benchmark

There is no csv.Sniffer benchmark in pyperformance, so I constructed a simple benchmark with:

import csv, pathlib, pyperf
def sniff(s): csv.Sniffer()._guess_delimiter(s, None)
r = pyperf.Runner()
sizes = [1024, 2048, 4096]
for file in pathlib.Path("/home/maurycy/CSVsniffer/CSV/").glob("*.csv"):
    for s in sizes:
        with file.open() as f:
            try:
                r.bench_func(f"csv_sniff({file.name}, {s})", sniff, f.read(s))
            except UnicodeDecodeError:
                pass

using all 149 files from CSVSniffer (MIT License), reading only the sample, as recommended in docs.python.org example. That's what real users do, too.

Results

Geometric mean: 1.50x faster

The full results:

• https://gist.github.com/maurycy/7ffaf9fce76eeddcc68910cda1627e9d
• https://github.com/maurycy/cpython-benchmarks/tree/main/csv-sniffer-counter-set

Environment

% ./python -c "import sysconfig; print(sysconfig.get_config_var('CONFIG_ARGS'))"
'--with-lto' '--enable-optimizations'

sudo ./python -m pyperf system tune ensured.

Notes

• The optimization is in csv.Sniffer()._read_delimiter() which runs only if regular expressions in csv.Sniffer()._guess_quote_and_delimiter() failed, so there's no guarantee that csv.Sniffer().sniff() will always be faster
• My original patch relied on confusing set operations.

• Issue: csv.Sniffer._guess_delimiter() iterates over all ASCII on each line #137627