Repository navigation
Casting float to int gives wrong number #5802
Description
CircuitPython version
Adafruit CircuitPython 7.1.0-rc.1 on 2021-12-25; Raspberry Pi Pico with rp2040Code/REPL
>>> 23.40 * 100
2340.0
>>> int(23.40 * 100)
2339Behavior
The floating point version is 2340.0 and when cast to int becomes 2339. This works fine in regular python:
Python 3.9.7 (default, Sep 10 2021, 14:59:43)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 23.40 * 100
2340.0
>>> int(23.40 * 100)
2340Description
Without knowing the implementation details, the problem reminds me of issues I've seen related to IEEE 754 elsewhere.
Additional information
No response
Activity
dhalbert commentedon Dec 31, 2021
2340.0 cannot be represented exactly with the bits we have. See https://www.h-schmidt.net/FloatConverter/IEEE754.html
In general one should round, not truncate.
Note that the float representation we use drops the two lowest bits as well: it's a 30-bit float with two bits less of mantissa. There is more discussion of this kind of thing in the MicroPython issues, for instance: micropython#4212.
georgboe commentedon Dec 31, 2021
Ok, thanks for explaining Dan 👍
I will close the issue.