WIP: ENH: autodecode pandas timestamps #10638

jklymak · 2018-03-01T17:34:26Z

PR Summary

attempt at #10533 after @efiring 's suggestion.

The following now works. (needs a recent pandas for the deregister step to work).

ping @jorisvandenbossche - if you have a more complete or canonical PR, feel free to push instead of this....

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

pd.plotting.deregister_matplotlib_converters()

xticks = pd.date_range(start="10/10/2017", end="14/10/2017",
                       freq='D')
print(type(xticks))
print(xticks.to_pydatetime())
plt.plot(xticks, np.arange(len(xticks)))
plt.xticks(xticks, xticks.strftime("%a %m-%d"))
plt.show()

PR Checklist

Has Pytest style unit tests
Code is PEP 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

anntzer · 2018-03-01T17:57:48Z

lib/matplotlib/dates.py

@@ -430,6 +430,9 @@ def date2num(d):
    For details see the module docstring.
    """

+    if hasattr(d, "to_pydatetime"):


add a comment that this is to give pandas a special treatment here? (and below)

efiring · 2018-03-01T18:03:21Z

lib/matplotlib/dates.py

@@ -430,6 +430,9 @@ def date2num(d):
    For details see the module docstring.
    """

+    if hasattr(d, "to_pydatetime"):
+        d = d.to_pydatetime()


I think we are unnecessarily losing efficiency here; to avoid that, use the same 2-part conditional here as you use in get_converter. That way, if the input is basically datetime64 it can be handled by your nice, efficient, fully vectorized converter instead of making a round-trip through the clunky datetime.datetime machinery.

Can pandas go to datetime64? Ity my understanding that the underlying data type is datetime64[ns], but I don't know what attribute to test for to get that to come out...

Oh, duh, I see...

jorisvandenbossche

Thanks for this!
I don't have a PR or branch yet (was planning to do that this evening). I was thinking to go the way to check for the dtype, but anyway, I think there are multiple options that are not necessarily better or worse.

jorisvandenbossche · 2018-03-01T18:45:32Z

lib/matplotlib/dates.py

+    if hasattr(d, "values"):
+        d = d.values
+    if hasattr(d, "to_pydatetime"):
+        d = d.to_pydatetime()


I am not sure this one is needed, as date2num already handles Timestamp objects (them being datetime.datetime subclasses):

In [82]: date2num(pd.Timestamp('2012-01-01')) Out[82]: 734503.0 In [85]: date2num(datetime.datetime(2012, 1, 1)) Out[85]: 734503.0

efiring · 2018-03-01T21:35:13Z

lib/matplotlib/units.py

@@ -154,6 +154,10 @@ def get_converter(self, x):
        # DISABLED cached = self._cached.get(idx)
        # DISABLED if cached is not None: return cached

+        if hasattr(x, "values"):
+            # this unpacks pandas datetime objects...


It is actually much more general than that; it is returning the underlying ndarray from a pandas Series or DataFrame.

efiring

Let's give it a try.

jklymak · 2018-03-01T22:50:41Z

I do wonder if this should be in 2.2, but I guess pandas is eager to drop matplotlib as an unconditional dependency?

efiring · 2018-03-01T23:00:41Z

It changes behavior for base mpl only in the case where an object with a ".values" attribute is supplied. The only danger I see there is that someone might have an ndarray subclass with a ".values" attribute that does something other than return the ndarray. So, it's fairly safe for mpl. It could be made even safer by checking to ensure .values really has returned an ndarray, but I doubt this will be necessary, at least for now.

I can imagine that it might have an undesired effect when pandas is in use, by always calling .values instead of simply passing the pandas object on, possibly to be handled by a registered converter that would be looking for the original object. It would be good to have a pandas person look at it from that standpoint. I don't know whether pandas does, or would, register such a converter.

jklymak · 2018-03-01T23:15:39Z

Oooops, yes, the old version will bypass the pandas converter, beacuse I put the check in the wrong spot (not where you told me to). I think that was a mistake born of another mistake.

This newly pushed version still works w/ the above example, and has the converter after pandas should have chosen a converter...

jorisvandenbossche · 2018-03-01T23:33:55Z

I will take a closer look tomorrow from the pandas side. But, if pandas registers converters, it registers the same converters for both pandas.Timestamp as datetime.datetime and np.datetime64. So I don't think we have to worry about bypassing the pandas converter, if it is registered.

It might be good to add a few tests for this (I suppose you can directly use pandas there to actually test it?)

jklymak · 2018-03-01T23:36:23Z

Yes, we can use pandas, though test suggestions welcome.

jklymak · 2018-03-04T00:28:22Z

@jorisvandenbossche Though this has been merged, it really should get some tests. happy to help with a PR to get those in.

Backport PR #10638 on branch v2.2.x

jklymak added this to the v2.2.0 milestone Mar 1, 2018

jklymak added third-party integration topic: date handling topic: units and array ducktypes labels Mar 1, 2018

anntzer reviewed Mar 1, 2018

View reviewed changes

efiring reviewed Mar 1, 2018

View reviewed changes

jklymak force-pushed the enh-autodecode-pandastimestamp branch from caa6de5 to 333b2ea Compare March 1, 2018 18:37

jorisvandenbossche reviewed Mar 1, 2018

View reviewed changes

jklymak force-pushed the enh-autodecode-pandastimestamp branch 2 times, most recently from 752e403 to 16ac8b7 Compare March 1, 2018 20:26

efiring reviewed Mar 1, 2018

View reviewed changes

jklymak force-pushed the enh-autodecode-pandastimestamp branch from 16ac8b7 to cecf1f5 Compare March 1, 2018 22:29

efiring approved these changes Mar 1, 2018

View reviewed changes

jklymak force-pushed the enh-autodecode-pandastimestamp branch from cecf1f5 to 873addf Compare March 1, 2018 23:14

ENH: autodecode pandas timestamps

59bbccb

jklymak force-pushed the enh-autodecode-pandastimestamp branch from 873addf to 59bbccb Compare March 1, 2018 23:17

tacaswell merged commit ea02861 into matplotlib:master Mar 3, 2018

lumberbot-app bot pushed a commit that referenced this pull request Mar 3, 2018

Backport PR #10638: WIP: ENH: autodecode pandas timestamps

e5fc2ac

lumberbot-app bot mentioned this pull request Mar 3, 2018

Backport PR #10638 on branch v2.2.x #10663

Merged

tacaswell added a commit that referenced this pull request Mar 4, 2018

Merge pull request #10663 from matplotlib/auto-backport-of-pr-10638

037b176

Backport PR #10638 on branch v2.2.x

tacaswell mentioned this pull request Mar 4, 2018

Recognize pandas Timestamp objects for DateConverter? #10533

Closed

jklymak added the pandas integration label Jul 15, 2018

This was referenced Jul 15, 2018

Use data argument for scatter plotting timestamps from pandas #11391

Closed

FIX: clean up unit conversion unpacking of data, particularly for dates and pandas series #11664

Closed

jklymak deleted the enh-autodecode-pandastimestamp branch July 16, 2018 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: ENH: autodecode pandas timestamps #10638

WIP: ENH: autodecode pandas timestamps #10638

jklymak commented Mar 1, 2018

anntzer Mar 1, 2018

efiring Mar 1, 2018

jklymak Mar 1, 2018

jklymak Mar 1, 2018

jorisvandenbossche left a comment

jorisvandenbossche Mar 1, 2018

efiring Mar 1, 2018

jklymak Mar 1, 2018

efiring left a comment

jklymak commented Mar 1, 2018

efiring commented Mar 1, 2018

jklymak commented Mar 1, 2018

jorisvandenbossche commented Mar 1, 2018

jklymak commented Mar 1, 2018

jklymak commented Mar 4, 2018

WIP: ENH: autodecode pandas timestamps #10638

WIP: ENH: autodecode pandas timestamps #10638

Conversation

jklymak commented Mar 1, 2018

PR Summary

PR Checklist

anntzer Mar 1, 2018

Choose a reason for hiding this comment

efiring Mar 1, 2018

Choose a reason for hiding this comment

jklymak Mar 1, 2018

Choose a reason for hiding this comment

jklymak Mar 1, 2018

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche Mar 1, 2018

Choose a reason for hiding this comment

efiring Mar 1, 2018

Choose a reason for hiding this comment

jklymak Mar 1, 2018

Choose a reason for hiding this comment

efiring left a comment

Choose a reason for hiding this comment

jklymak commented Mar 1, 2018

efiring commented Mar 1, 2018

jklymak commented Mar 1, 2018

jorisvandenbossche commented Mar 1, 2018

jklymak commented Mar 1, 2018

jklymak commented Mar 4, 2018