I can reproduce that, but it works fine with a much more modest chunk size of 2000. By chunking with N larger than your data set, you are basically not chunking.

I'll be honest, I don't know why you get an overflow. The limit is somewhere near 29350 on my machine before I have to chunk. I am somewhat suspicious we have a short-int sized buffer in there somewhere, which seems unnecessary, but maybe that is an agg limitation. No doubt @anntzer will tell us that cairo has no such limitation :-)

I think(?) getting rid of that limit is just a matter of

diff --git i/src/_backend_agg.cpp w/src/_backend_agg.cpp
index 6ae11167e..565cc9a62 100644
--- i/src/_backend_agg.cpp
+++ w/src/_backend_agg.cpp
@@ -45,7 +45,7 @@ RendererAgg::RendererAgg(unsigned int width, unsigned int height, double dpi)
       rendererBase(),
       rendererAA(),
       rendererBin(),
-      theRasterizer(8192),
+      theRasterizer(-1),
       lastclippath(NULL),
       _fill_color(agg::rgba(1, 1, 1, 0))
 {

(the argument is an unsigned upper bound, so -1 means "until ints overflow" :)) but of course that means that you'll run out of memory, instead of cleanly raising an exception (much) earlier, if you try to draw something too big. (Perhaps that's better?)

Or raise the limit given that it is likely tailored to relatively old machines? Like we don't want someone to thrash, but 100,000 data points seems small?

Agreed, but I'm not going to pick a limit for you here :)

Unfortunately I have no idea what that number means. Agg isn't very well documented so far as I can tell...

I assume it's the (max)size of some internal rasterization buffer, but I don't know either.

Running out of memory is definitely worse.

This is documented in an odd place: https://matplotlib.org/tutorials/introductory/usage.html#splitting-lines-into-smaller-chunks .

When we draw a Line2D eventually that falls through to render.draw_path(...). Due to details of Agg, there is a maximum number of vertices that we can feed to it (and I believe there is something in the draw process who's run time scales badly with the number of vertices we are feeding it).

An additional wrinkle is that the number of of verities that we feed the renderer is actually data dependent as internally the (x, y) data you provide is simplified to remove redundant vertices (also documented in a surprising place : https://matplotlib.org/tutorials/introductory/usage.html#line-segment-simplification). Thus if you change the data to

x = np.linspace(0, 100, N:=100000)
y = np.ones(N)

it saves both fine because the even though you are giving Matplotlib 100k points, internally we are probably only passing 2 points into the rasterizer (way down at the bottom). This also means that for "smooth" data, we get some automatic smart down sampling (e.g. np.sin(x) also works fine with this data because it is locally it is close enough to linear that the curve gets simplified to something that fits in to the renderer). In that sense the test data here is the worst-case scenario as it in general never goes in the same direction.

As of 2.1 (via #8776) we picked up "back tracking" simplification as well (so if you have data that is just going up and down in a column of the rendered image we only only keep the vertices at the top and bottom. In the normal case that is enough to us under the threshold, but not for the 500dpi case.

Tweaking the simplification threshold to be higher e.g.

matplotlib.rcParams['path.simplify_threshold'] = 1

also makes the noise example render (and it works down to 0.5 ish).

In #4404 @anntzer suggested making the chunk size automatic (e.g. it fails, we reduce it and try again until it works), but I was skeptical then and more skeptical now given that the simplification logic is another knob to reach for.

At a minimum I am going to put in a PR making that error message give better direction as to how to fix the problem!

I don't think allowing memory to fill is a good idea. OTOH if we can allow larger chunks because on-board memory has increased an order of magnitude, I dont see why we should burden users with twiddling anything. 100,000 points doesn't sound like a very big dataset in 2021.

The issue isn't the number of points in the dataset it is how much irreducible texture it has.

I chased this through and it is an upper bound

I have a patch to make the error messages more actionable and am becoming convinced than a 8-16x bump on the max is in order.

One more comment while testing this, if we make the size a bit bigger we can get a 100k pathological data set to draw at 400dpi, but by setting the chunk_size to 10k we get a ~5k speed up in draw time.

Process Python finished
Python 3.10.0a4+ (heads/main:d16f6176ab, Jan  9 2021, 17:51:02) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: import io

In [3]: import numpy as np

In [4]: from matplotlib.figure import Figure

In [5]: buff = io.BytesIO()

In [6]: fig = Figure()

In [7]: ax = fig.subplots()

In [8]: points = np.ones(100_000)

In [9]: points[::2] *= -1

In [10]: ax.plot(points)
Out[10]: [<matplotlib.lines.Line2D at 0x7f9304728a00>]

In [11]: fig.savefig(buff, format='png', dpi=400)

In [12]: fig.savefig(buff, format='png', dpi=400)

In [13]: %timeit fig.savefig(buff, format='png', dpi=400)
7.27 s ± 17.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [14]: from matplotlib import (
     ...     collections, path, pyplot as plt, transforms as mtransforms, rcParams)

In [15]: rcParams['agg.path.chunksize'] = 10_00

In [16]: %timeit fig.savefig(buff, format='png', dpi=400)
1.28 s ± 11.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [17]: 

So I think the main trade off on upping the chunk limit is less memory usage, and more run time. With a good error message we can hopefully direct the users who run into this to the chunk size with a sense of what "good" values are and save them run time. Where as if we push the limit higher and higher we are going to get bug reports of "Matplotlib is slow" which is harder to identify and point people to solutions.

Another option could be to set the default chunk size to be like 10k or 50k. That would, with an upped limit, be enough to have saved the OP out of the box and should be a no-op for a vast majority of users (who I assume are mostly plotting things shorter than 50k).

Is there a way to check how much memory each case is using? The 8192 number seemed ridiculously small, but I don't know how that corresponded to a memory.

To quote myself from 2014 #3777 (comment)

I am happy enough with this and I suspect getting any happier would involve spending a week doing nothing but reading the Agg code (and get worse before it got better) so 👍 from me.

I'm not fully sure what "cells" in the context are. But observational there is some super-linear scaling in the rasterization time with the number of cells (I suspect quadratic as I think each cell requires you to scan all of the other cells based on vague memories of conversations with Mike a few years ago). I am starting to think that the original limit was more about run time rather than memory usage (which is probably where we should have started given that the chunksize in documented under a "performance" heading).