| Re: fix(4.70*100) is... 469 [message #53639 is a reply to message #53547] |
Thu, 19 April 2007 12:06  |
![Paul Van Delst[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Paul Van Delst[1] is currently offline")
Paul Van Delst[1]
Messages: 1157
Registered: April 2002
|
Senior Member |
|
|
b_efremova@yahoo.com wrote:
> I'll try one more time , see if it works.
>
> Here is my initial statement:
> When I handle a floating point number, 470.000
> it better stay the same number with the precision I use it.
>
> It may well be 469.99999999999999999999999 and this is
> perfectly fine with me if when used with a floating point precision
> it is rounded to 470.000
>
> OK, so if I do something, like
> converting 470.00 to double, I have no right to complain when
> the number I get is 469.99999999999999435462346 with any
> arbitrary numbers in the digits exceeding the precision I had
> provided.
>
> On the other side, I would expect when I use this number with the
> provided precision or lower to matter if I had given the number
> 470.000 or 469.999.
What do *you* mean by "provided precision"?
> In short, when converting my number to something of lower precision
> like integer
> I would expect the number to be rounded.
What you expect and what actually happens is demonstrably (and reproducably) different.
There are several ways to "convert" a floating point number to an integer. Rounding is
just one way.
> but the integer of 469.9999 is 469. which is not true.
>
> I also had the wrong assumption that FIX will FIRST round the number
> to the precision I'm
> working with and THEN truncate.
> Now when I think of this, it is not very intellicgent assumption.
> Well, I did it.
Ah, well. We are now in the realm of the lessons learnt in the school of hard knocks. :o)
It can be a royal pain in the rear end, but it can be good way of learning things -- we
tend not to forget the associated blood, sweat and tears.
> On the other hand I stilll think that
>
> print,469.9999,format='(i3)'
> should be
> 470
>
> but it is 469 insted.
> What is wrong about this assumption of mine?
It's simply another syntax for
IDL> print, INT(469.999)
469
Computers have zero intelligence - you have to ask them to do *exactly* what you want.
There's no way for the circuitry to divine that while you actually said something like
INT(469.999)
you really meant
ROUND(469.999)
(or vice versa)
Many a program has crashed in a big hairy heap on the floor because of this type of
assumption.
Have a read of:
http://docs.sun.com/source/806-3568/ncg_goldberg.html
and check out
http://tinyurl.com/2wnggj
from your library.
cheers,
paulv
--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
