I find it pretty dangerous that depending on when you set epoch you may get totally different outcomes and potentially inconsistent results. Especially if you think about some package that you import setting the epoch - a nightmare to debug.

You shouldn't get "totally different outcomes", unless you mix datetime with floats in your code (and you change the epoch). Of course you get different outcomes if one of them is broken, but thats what this is meant to fix

You can always check the epoch with mdates.get_epoch() if something funky starts going on.

If we want sub-millisecond plotting (which you kind of want for zooming) then this is the way to do it.

The ultimate problem is that the original choice was to make the epoch year 0001 which is too far away for modern dates to get much floating point resolution.

Clearly, it's not the way to do it; it's one way to do it. Pandas uses a totally different approach -which is also non-ideal and subject to a lot of confusion. But e.g. the case from above works out of the box in pandas.

import numpy as np 
import pandas as pd
import matplotlib.pyplot as plt

x = np.arange('2000-01-01T00:00:00.0', '2000-01-01T00:00:00.000100',
              dtype='datetime64[us]')
y = np.arange(0, len(x))

df = pd.DataFrame({"XX" : x, "YY" : y})
fig, ax = plt.subplots(constrained_layout=True)
df.plot(x="XX", y="YY")
plt.show()

Concerning totally different outcomes, what I mean is that you may e.g. (re)set epoch too early, like

x = np.arange('2000-01-01T00:00:00.0', '2000-01-01T00:00:00.000100',
              dtype='datetime64[us]')
y = np.arange(0, len(x))

mdates.set_epoch('1999-01-01')
fig, ax = plt.subplots(constrained_layout=True)
ax.plot(x, y)
ax.xaxis.set_major_formatter(mdates.DateFormatter("%H:%M:%S.%f\n%Y-%m-%d"))

mdates.reset_epoch()

plt.show()

resulting in the axes showing dates from when Jesus was born,

In this case it's rather obvious, but as said, some module you import in cell 32 of your notebook may (re)set epoch and it would still affect the plot you started in cell 2 and show in cell 35.

Similarly you may read in some data without having set an epoch, and only set it later before plotting,

x,y = np.loadtxt(..., 
                 converters={0: mdates.datestr2num})

#lots of other stuff

mdates.set_epoch('1999-01-01')
plt.plot(x,y)

I had the same thought as @ImportanceOfBeingErnest walking home from the train, probably need to bind the epoch to the axes some how?

x,y = np.loadtxt(..., 
                 converters={0: mdates.datestr2num})

#lots of other stuff

mdates.set_epoch('1999-01-01')
plt.plot(x,y)

is an interesting one. I didn't know people did that. Binding the epoch to the axes won't help there (and is in fact why the epoch needs to be a global). Is there an advantage to the user using datestr2num instead of one of the standard libraries and keeping their dates as datetime?

What does Pandas do?

I guess one approach would be to sentinel the epoch so that once its used anywhere, set_epoch throws an error (or a warning and doesn't change the epoch). ie. you can really only call it before you use anything from dates.py.

I guess also to be clear, this is really not meant to be used willy-nilly. But we do get issues where people complain about microseconds not plotting correctly, and its kind of a shame because we have chosen a poor epoch for 99% of the people who want microsecond accuracy (not many folks were tracking microseconds back in year 0001). I really expect this to be a seldom used feature.

In #7138 (comment) I suggested making this callable only once, and only before any call to date2num (if a library calls it before you, that's too bad -- we could even explicitly discourage other libraries to call it, suggesting that the use should be reserved to end users). Perhaps having a per-axes epoch would be even better, but that's likely much harder to do properly...

Perhaps having a per-axes epoch would be even better, but that's likely much harder to do properly...

If you go that way, probably a per-axis epoch ...

A sentinel is easy enoiugh except it makes writing the tests and docs really hard because I need to change the epoch which then would expose to users the private method to reset the epoch.

A per-axis epoch is basically a new locator/formatter. Then the date2num etc could go be methods on the locator as would the epoch. We could easily go that route but then the new locators are not backwards compatible with the old ones and we have the rigamarole of making the new ones standard. Ie I vastly prefer ConciseDateFormatter to the default AutoDateFormatter, but it’s hard to see how we will ever make it the default.

The tests could use a private method to reset it, I agree it's less nice for the docs :/

This new version has a sentinel. So:

import matplotlib.date as mdate
import numpy as np 
import matplotlib.pyplot as plt 

x = np.arange('2000-01-01T00:00:00.0', '2000-01-01T00:00:20.000100',
              dtype='datetime64[s]')
y = np.arange(0, len(x))
x2 = np.arange('2001-01-01T00:00:00.0', '2001-01-01T00:00:20.000100',
              dtype='datetime64[s]')
y2 = np.arange(0, len(x))
fig, ax = plt.subplots()
ax.plot(x, y)
mdates.set_epoch('1990-01-01')
ax.plot(x2, y2)

fails w/

  File "testepoch.py", line 13, in <module>
    mdates.set_epoch('1990-01-01')
  File "/Users/jklymak/matplotlib/lib/matplotlib/dates.py", line 253, in set_epoch
    raise RuntimeError('set_epoch must be called before dates plotted.')
RuntimeError: set_epoch must be called before dates plotted.

The docs need to call _reset_epoch, and just have a huge caveat about doing so.

ping @efiring, as having some interest in dates

There is a lot here, and I appreciate the work. I am sympathetic to the idea of providing a way to retreat from the unfortunate choices of epoch used by mpl and by Matlab, for that matter. This is a bit like a convention that I started very long ago for my work with shipboard ADCP data: using floating-point time in days (which I call "decimal days") relative to the start of a specified year (which I call the "yearbase"). It also fits in with many satellite-based data products which typically give time in days since the start of 1950. Other data products use other years. Unix uses seconds since the start of 1970. In my understanding, the "epoch" in such a case is the start of the specified year, so noon on January 1 of the epoch year is 0.5 days. I think this is consistent with the usage in https://en.wikipedia.org/wiki/Epoch, for example.

In mpl datenums, noon on January 1 of the year 1 is 1.5, so I would have to say that the real epoch for mpl is December 31 on 1 BCE (since there is no year zero). But this conflicts with the way you are using "epoch", claiming that the mpl default is January 1 of the year 1. This is a recipe for confusion.

Putting aside the question of what the epoch really is, some of the motivations for this change seem contrived, though. Is there a real use case for zooming from dates to microseconds, especially given that all of the date-time formats in question are locally incorrect at the 1-second level because of leap-seconds?

My experience with my decimal day and yearbase also makes me cautious here. I found that I had to be very careful to keep track of the yearbase, adding complexity to the code. There were valid reasons for the design when it was developed, but if I were starting now I would probably pick a single epoch and stick with it.

Well the other application is people who have floats in seconds since 1970 (for instance). In order to get in mTpltolib yeardays they have to use datetime anyways to figure out how many days to add to that number. Or, we could just do it for them by letting them set the epoch. Or the example you just gave of yeardays 2017. Right now they need to know the offset

Discussed this briefly on the 1 Oct call. This seems a reasonable way forward.

While I agree that going to sub-seconds looses absolute accuracy, its the precision you usually want at such small scales.

I don't think the test failures are from this PR, but could be misunderstanding...

Putting back in draft mode to better understand #15008 (comment)

Why does the first plot in the example get broken Axis?

@QuLogic The axis in that example is one of the things this PR is addressing. It gets broken by roundoff error. Please see the example at the top of this PR as well

Rebased. Thanks for everyone's help with this!

🎉 Thank you everyone!