Skip to content

Trailing .0 in float literal can cause rounding error #5831

Closed
@dlech

Description

@dlech

I ran into this issue when formatting tests using black. It doesn't allow floating point numbers with only a trailing . it always adds a 0 after the ..

When this

print(int(14187745.))
print("%d" % 14187745.)

was changed to this

print(int(14187745.0))
print("%d" % 14187745.0)

it started failing on qemu-arm. It turns out the trigger is that it is 32-bit floating point and 14187745 has an exact decimal representation but 141877450 does not.

The error comes from mp_parse_num_decimal() where the number is multiplied by 10 for each trailing 0 after the decimal, then divided down back to the original number later.

dec_val = 10 * dec_val + dig;

Would it be too much of a performance hit to check that all digits after . are 0? Or perhaps just test for the case where there is only one digit after . and that it is 0?

Originally posted by @dlech in #4212 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    py-coreRelates to py/ directory in source

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions