Skip to content

PERF: to_datetime doesn't recognise Arrow date time dtypes and converts them again #52545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 of 3 tasks
phofl opened this issue Apr 8, 2023 · 0 comments · Fixed by #53301
Closed
2 of 3 tasks

PERF: to_datetime doesn't recognise Arrow date time dtypes and converts them again #52545

phofl opened this issue Apr 8, 2023 · 0 comments · Fixed by #53301
Assignees
@lithomas1
Labels
Arrow pyarrow functionality Datetime Datetime data dtype Performance Memory or execution speed performance

Comments

@phofl
Copy link
Member

phofl commented Apr 8, 2023 •
edited
Loading

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

In [3]: dr = pd.date_range("2019-12-31", periods=1_000_000, freq="s").astype(pd.ArrowDtype(pa.timestamp(unit="ns")))

In [4]: %timeit pd.to_datetime(dr)
1.84 s ± 8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

this should be instant and not take close to 2 seconds

cc @MarcoGorelli

Installed Versions

main

Prior Performance

No response
@phofl phofl added Performance Memory or execution speed performance Needs Triage Issue that has not been reviewed by a pandas team member Datetime Datetime data dtype labels Apr 8, 2023
@lithomas1 lithomas1 added Arrow pyarrow functionality and removed Needs Triage Issue that has not been reviewed by a pandas team member labels May 3, 2023
@lithomas1 lithomas1 mentioned this issue May 8, 2023
Merged
5 tasks
@lithomas1 lithomas1 self-assigned this May 8, 2023
@lithomas1 lithomas1 mentioned this issue May 19, 2023
Merged
5 tasks
This was referenced Feb 29, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lithomas1 lithomas1

Labels
Arrow pyarrow functionality Datetime Datetime data dtype Performance Memory or execution speed performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: to_datetime re-parsing Arrow-backed objects lithomas1/pandas
2 participants
@lithomas1 @phofl