Bug 1059 : Multiplying floats returns erroneous values.
Last modified: 2008-11-24 20:34




Status:
RESOLVED
Resolution:
INVALID -
Priority:
P2
Severity:
normal

 

Reporter:
nullkraft
Assigned To:
fry

Attachment Type Created Size Actions

Description:   Opened: 2008-11-24 20:09
I'm using Processing 0157 for Linux, Intel X86, and I have NOT linked to
another Java. The problem I see is with the following code:

float digits = 1132.0 * 0.0001;
text("Digits: " + digits, 50, 50);

Which displays :

Digits: 0.113199994

which is incorrect, but when I invert 0.0001 and divide by 10000 instead I
get the following :

1132.0 / 10000 = 0.1132

which is correct. They are both technically the same operation and should
have the same results. I tested other values from 1119.0 through 1132.0
with some values returning incorrect results and others returning the
correct results. I also tried :

1132 * 0.8356 = 945.89923 which should have been :

945.8992
Additional Comment #1 From fry 2008-11-24 20:15
floats are 32 bit floating point numbers. they are not perfect, they are
approximations. they offer around 4 to 6 digits of accuracy depending on
the case in which they're used, so you're going to get poor results if you
multiply a smallish number by 0.0001.

use a double if you need better numerical accuracy, then convert to float
before drawing since none of the APIs use doubles.
Additional Comment #2 From fry 2008-11-24 20:16
*** Bug 1058 has been marked as a duplicate of this bug. ***
Additional Comment #3 From nullkraft 2008-11-24 20:34
Actually, I thought about that too. But after testing larger/smaller
sequential numbers and getting the correct results I came to the conclusion
that there was something amiss. I don't know that it's even in Processing,
maybe it's in Java. I just thought I'd post it in case there's a real
problem. The fact that I can get the correct results by inverting and
multiplying is more than sufficient for my needs.