| Re: fix(4.70*100) is... 469 [message #53554 is a reply to message #53552] |
Thu, 19 April 2007 08:21   |
Christopher Thom
Messages: 66
Registered: October 2006
|
Member |
|
|
Quoth b_efremova@yahoo.com:
> Sorry Guys, I should have made myself clearer.
> I'm afraid David this is not actually the question you describe in
> your article.
> and I do not expect better accuracy than I provide.
>
> There is nothing wrong here with the floating point accuracy.
> print,4.700*100.00
> 470.000
>
> It is the conversion to integer (I imagine) which makes no sence.
>
> print,fix(4.700*100.00)
> 469
> also (which is what I really needed)
No. Read the article again...and the one on double precision...it is
exactly what is described there. You have provided IDL with a number that
has 8 decimal places of precision. 4.7 is really somewhere between
4.6999999 - 4.7000001, but cannot be precisely represented. i.e.
IDL> print,4.7
4.70000
IDL> print,4.7,f='(f18.16)'
4.6999998092651367
The important point is that converting the *actual number as represented
in the computer* to an integer, is NOT converting the number you *think*
is represented in the computer.
So...if you take the number that is actually in IDL...move the decimal
place 2 places to the right, you get
IDL> print,4.7*100,f='(f18.14)'
469.99996948242188
Now chop off every thing after the decimal place (which is what fix()
does)...and 469 is a prefectly reasonable answer to the question you
asked. If you want a better answer, you need to ask a better question :-)
I can't speak as to exactly how the conversion to integers happens within
the string() command you gave, but I imagine it's probably the same.
cheers
chris
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
