| Re: fix(4.70*100) is... 469 [message #53547 is a reply to message #53544] |
Thu, 19 April 2007 10:44   |
b_efremova@yahoo.com
Messages: 18
Registered: July 2005
|
Junior Member |
|
|
I'll try one more time , see if it works.
Here is my initial statement:
When I handle a floating point number, 470.000
it better stay the same number with the precision I use it.
It may well be 469.99999999999999999999999 and this is
perfectly fine with me if when used with a floating point precision
it is rounded to 470.000
OK, so if I do something, like
converting 470.00 to double, I have no right to complain when
the number I get is 469.99999999999999435462346 with any
arbitrary numbers in the digits exceeding the precision I had
provided.
On the other side, I would expect when I use this number with the
provided precision or lower to matter if I had given the number
470.000 or 469.999.
In short, when converting my number to something of lower precision
like integer
I would expect the number to be rounded.
but the integer of 469.9999 is 469. which is not true.
I also had the wrong assumption that FIX will FIRST round the number
to the precision I'm
working with and THEN truncate.
Now when I think of this, it is not very intellicgent assumption.
Well, I did it.
On the other hand I stilll think that
print,469.9999,format='(i3)'
should be
470
but it is 469 insted.
What is wrong about this assumption of mine?
Cheers
Boryana
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
