| Re: fix(4.70*100) is... 469 [message #53548 is a reply to message #53547] |
Thu, 19 April 2007 10:03   |
b_efremova@yahoo.com
Messages: 18
Registered: July 2005
|
Junior Member |
|
|
Thanks Chris,
I did,
I'll try a better quesstion then.
I totally agree that 470 is something between 469.9999 something and
470.00000 something
Here comes my question then:
print,469.9999,format='(f9.3)'
470.000
....
print,469.9999,format='(f9.0)'
470.
print,469.9999,format='(i3)'
469.
So converting to integer DOES NOT keep the number the same with the
give accuracy.
Instead it just truncates.
Sorry it took me so long to explain myself.
And thanks Paolo, I use round, and all works fine.
Cheers
Boryana
Christopher Thom wrote:
> Quoth b_efremova@yahoo.com:
>
>> Sorry Guys, I should have made myself clearer.
>> I'm afraid David this is not actually the question you describe in
>> your article.
>> and I do not expect better accuracy than I provide.
>>
>> There is nothing wrong here with the floating point accuracy.
>> print,4.700*100.00
>> 470.000
>>
>> It is the conversion to integer (I imagine) which makes no sence.
>>
>> print,fix(4.700*100.00)
>> 469
>> also (which is what I really needed)
>
> No. Read the article again...and the one on double precision...it is
> exactly what is described there. You have provided IDL with a number that
> has 8 decimal places of precision. 4.7 is really somewhere between
> 4.6999999 - 4.7000001, but cannot be precisely represented. i.e.
>
> IDL> print,4.7
> 4.70000
> IDL> print,4.7,f='(f18.16)'
> 4.6999998092651367
>
> The important point is that converting the *actual number as represented
> in the computer* to an integer, is NOT converting the number you *think*
> is represented in the computer.
>
> So...if you take the number that is actually in IDL...move the decimal
> place 2 places to the right, you get
>
> IDL> print,4.7*100,f='(f18.14)'
> 469.99996948242188
>
> Now chop off every thing after the decimal place (which is what fix()
> does)...and 469 is a prefectly reasonable answer to the question you
> asked. If you want a better answer, you need to ask a better question :-)
>
> I can't speak as to exactly how the conversion to integers happens within
> the string() command you gave, but I imagine it's probably the same.
>
> cheers
> chris
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
