| Re: fix(4.70*100) is... 469 [message #53629 is a reply to message #53547] |
Thu, 19 April 2007 23:43   |
Qing
Messages: 12
Registered: February 2007
|
Junior Member |
|
|
On Apr 20, 3:44 am, b_efrem...@yahoo.com wrote:
> I'll try one more time , see if it works.
>
> Here is my initial statement:
> When I handle a floating point number, 470.000
> it better stay the same number with the precision I use it.
>
> It may well be 469.99999999999999999999999 and this is
> perfectly fine with me if when used with a floating point precision
> it is rounded to 470.000
>
> OK, so if I do something, like
> converting 470.00 to double, I have no right to complain when
> the number I get is 469.99999999999999435462346 with any
> arbitrary numbers in the digits exceeding the precision I had
> provided.
>
> On the other side, I would expect when I use this number with the
> provided precision or lower to matter if I had given the number
> 470.000 or 469.999.
>
> In short, when converting my number to something of lower precision
> like integer
> I would expect the number to be rounded.
>
> but the integer of 469.9999 is 469. which is not true.
>
> I also had the wrong assumption that FIX will FIRST round the number
> to the precision I'm
> working with and THEN truncate.
> Now when I think of this, it is not very intellicgent assumption.
> Well, I did it.
> On the other hand I stilll think that
>
> print,469.9999,format='(i3)'
> should be
> 470
>
> but it is 469 insted.
> What is wrong about this assumption of mine?
> Cheers
> Boryana
Definately "sky is falling" introduces great fun and interests to us
scientists! To continue ...
(1) It is not a problem about just the FIX function:
IDL> print, floor(4.7*100)
469
(2) Can we always use ROUND instead of FIX or FLOOR? Then why the hell
to have FIX and FLOOR to get confused?
(3) A precision issue? definitely need to read again at
http://www.dfanning.com/math_tips/sky_is_falling.html .
"There's nothing worse that trying to debug code and discovering weird
results are related to the precision of the represetation" - it can
also be fun!
But what about:
IDL> print, fix(100D*4.7) ============= it can still be argued
as a precision issue as long as you use 4.7 as a example !!!
469
(4) "... maybe apart from an insidious compiler bug, but that would
never happen with IDL!"
does the problem happen in just IDL (on Windows, Lenux, MaxOS, ...)?
Do we have the same problem in C/C++, FORTRAN or even BASIC?
Lets continue the hunt... it may not be just fun. Is is possible that
the difference between 469 and 470 could end up sending a satellite
off its track :-((
Really, I am serious! :-)))
Qing
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
