Skip to content

BUG: assert_frame_equal(check_dtype=False) fails when comparing two DFs containing pd.NA that only differ in dtype (object vs Int32) #61473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 of 3 tasks
michiel-de-muynck opened this issue May 21, 2025 · 6 comments · May be fixed by #61525
Open
2 of 3 tasks

BUG: assert_frame_equal(check_dtype=False) fails when comparing two DFs containing pd.NA that only differ in dtype (object vs Int32) #61473

michiel-de-muynck opened this issue May 21, 2025 · 6 comments · May be fixed by #61525
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. good first issue Testing pandas testing functions or related to the test suite

Comments

@michiel-de-muynck
Copy link
Contributor

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from pandas.testing import assert_frame_equal

df1 = pd.DataFrame(
    {
        "x": pd.Series([pd.NA], dtype="Int32"),
    }
)
df2 = pd.DataFrame(
    {
        "x": pd.Series([pd.NA], dtype="object"),
    }
)

assert_frame_equal(df1, df2, check_dtype=False) # fails, but should succeed

Issue Description

Output of the above example:

AssertionError: DataFrame.iloc[:, 0] (column name="x") are different

DataFrame.iloc[:, 0] (column name="x") values are different (100.0 %)
[index]: [0]
[left]:  [nan]
[right]: [<NA>]

When comparing DataFrames containing pd.NA using check_dtype=False, the test incorrectly fails despite the only difference being the dtype (Int32 vs object).

Note that the values in the dataframe really are the same:

print(type(df1["x"][0])) # prints <class 'pandas._libs.missing.NAType'>
print(type(df2["x"][0])) # prints <class 'pandas._libs.missing.NAType'>

Related issues:

Expected Behavior

The test should succeed, since the only difference is the dtypes, and check_dtype=False.

Installed Versions

pandas : 2.2.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 19.0.1
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.41
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None
@michiel-de-muynck michiel-de-muynck added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels May 21, 2025
@rhshadrach
Copy link
Member

Thanks for the report, this would pass if when converting the EA to a NumPy array we cast to object dtype. I haven't looked to see if this might cause issues in other cases. Since this is aimed at tests, I'm wondering if changing to object dtype is okay here.

cc @jbrockmendel @mroeschke for any thoughts.

@rhshadrach rhshadrach added Testing pandas testing functions or related to the test suite Needs Discussion Requires discussion from core team before further action ExtensionArray Extending pandas with custom dtypes or arrays. and removed Needs Triage Issue that has not been reviewed by a pandas team member labels May 21, 2025
@jbrockmendel
Copy link
Member

this would pass if when converting the EA to a NumPy array we cast to object dtype

Yah I'm pretty sure that the behavior of df1['x'].to_numpy() casting to a float dtype was a much-discussed intentional decision. Changing that would be a can of worms.

I'm inclined to just discourage the use of a) check_dtype=False and b) using pd.NA in an object dtype column (note that df1 == df2 raises)

@rhshadrach
Copy link
Member

rhshadrach commented May 23, 2025 •
edited
Loading

@jbrockmendel - sorry, I wasn't clear. I meant just inside assert_frame_equal to use .to_numpy(dtype="object") when check_dtype=False rather than just .to_numpy(). Agreed changing the behavior of .to_numpy() is off the table.

@jbrockmendel
Copy link
Member

Gotcha, fine by me

@rhshadrach rhshadrach added good first issue and removed Needs Discussion Requires discussion from core team before further action labels May 30, 2025
iabhi4 added a commit to iabhi4/pandas that referenced this issue May 31, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
3577b05 
…ndas-dev#61473)
@iabhi4 iabhi4 linked a pull request May 31, 2025 that will close this issue
Open
4 tasks
@venturero
Copy link

how can i make contribution to solve this, can you please give advice to me? @iabhi4 @rhshadrach

iabhi4 added a commit to iabhi4/pandas that referenced this issue Jun 1, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
1aff954 
…ndas-dev#61473)
iabhi4 added a commit to iabhi4/pandas that referenced this issue Jun 1, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
ad74d18 
…ndas-dev#61473)
@iabhi4
Copy link
Contributor

iabhi4 commented Jun 1, 2025

Hi @venturero

I already raised a PR for this based on the above discussion. You can checkout other issues from the Issues tab and follow the contribution guide to submit a clean fix for the issue you're tackling.

iabhi4 added a commit to iabhi4/pandas that referenced this issue Jun 1, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
e738953 
…ndas-dev#61473)
iabhi4 added a commit to iabhi4/pandas that referenced this issue Jun 3, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
5e422e4 
…ndas-dev#61473)
iabhi4 added a commit to iabhi4/pandas that referenced this issue Jun 3, 2025
@iabhi4
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
fcac82c 
…ndas-dev#61473)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. good first issue Testing pandas testing functions or related to the test suite
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

BUG: Fix assert_frame_equal dtype handling when check_dtype=False (#61473) iabhi4/pandas
5 participants
@michiel-de-muynck @jbrockmendel @rhshadrach @iabhi4 @venturero