Skip to content

[WIP] FIX Add tests for pyarrow dtypes in pandas Dataframes #26651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

[WIP] FIX Add tests for pyarrow dtypes in pandas Dataframes #26651

wants to merge 5 commits into from

Conversation

jovan-stojanovic
Copy link
Contributor

Co-authored-by: @LeoGrin

Reference Issues/PRs

Fixes partly #25896, see also #26464

What does this implement/fix? Explain your changes.

Adds tests when using the Arrow backend with pandas, for dataframes with arrow dtypes (added with Pandas 2.0)

Any other comments?

@github-actions
Copy link

github-actions bot commented Jun 21, 2023 •
edited
Loading

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 650b228

@Vincent-Maladiere
Copy link
Contributor

Vincent-Maladiere commented Jun 22, 2023 •
edited
Loading

Hey @LeoGrin and @jovan-stojanovic ! The tests are failing because pyarrow is missing in the test suite environment.

It has recently been added to the pylatest_conda_forge_mkl_linux-64 CI environment, but you want to add it to every other environment using pandas in update_environments_and_lock_files.py.

scikit-learn/build_tools/update_environments_and_lock_files.py

Lines 86 to 99 in 8c054b6

conda_build_metadata_list = [
{
"build_name": "pylatest_conda_forge_mkl_linux-64",
"folder": "build_tools/azure",
"platform": "linux-64",
"channel": "conda-forge",
"conda_dependencies": common_dependencies + [
"ccache",
"pytorch",
"pytorch-cpu",
"polars",
"pyarrow",
"array-api-compat",
],

For instance, the following environments failed in Azure CI:

  • py38_conda_forge_openblas_ubuntu_2204
  • pylatest_conda_forge_mkl_linux-64 (pyarrow has been added in main, so a rebasing on main + running the test one more time shouldn't fail)

Other environments that requires pyarrow:

  • pylatest_conda_forge_mkl_osx-64
  • pylatest_conda_mkl_no_openmp
  • pylatest_conda_forge_mkl_no_coverage
  • py38_conda_defaults_openblas
  • pylatest_pip_openblas_pandas

Or we could add pyarrow to the list common_dependencies_without_coverage and remove it from pylatest_conda_forge_mkl_linux-64.

What do the core-devs think?

@jovan-stojanovic jovan-stojanovic changed the title Add tests for pyarrow dtypes in pandas Dataframes [WIP] FIX Add tests for pyarrow dtypes in pandas Dataframes Jun 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jovan-stojanovic @Vincent-Maladiere @LeoGrin