| Re: fix(4.70*100) is... 469 [message #53549 is a reply to message #53548] |
Thu, 19 April 2007 10:00   |
mmeron
Messages: 44
Registered: October 2003
|
Member |
|
|
In article <1176993178.932710.145030@p77g2000hsh.googlegroups.com>, b_efremova@yahoo.com writes:
> Sorry Guys, I should have made myself clearer.
> I'm afraid David this is not actually the question you describe in
> your article.
> and I do not expect better accuracy than I provide.
>
> There is nothing wrong here with the floating point accuracy.
> print,4.700*100.00
> 470.000
>
> It is the conversion to integer (I imagine) which makes no sence.
>
> print,fix(4.700*100.00)
> 469
> also (which is what I really needed)
>
> print,string(4.700*100.00,format='(i3)')
> 469
>
Nah, it is a floating point accuracy issue. Remember, the output of
print is *not* necessarily the number stored in memory, it is just
said number rounded to some default number of decimal places (cost me
many sleepless nights, this one). So, indeed
IDL> print, 4.7*100
470.000
But, lets take a look at more decimal places
IDL> print, 4.7*100, format = '(f12.8)'
469.99996948
Since FIX always picks the integer part throwing away the fraction
(i.e. always rounds down) this becomes 469. You'll avoid this problem
if you'll use ROUND instead of FIX.
Mati Meron | "When you argue with a fool,
meron@cars.uchicago.edu | chances are he is doing just the same"
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
