| Re: fix(4.70*100) is... 469 [message #53626 is a reply to message #53551] |
Fri, 20 April 2007 01:04   |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
David Fanning wrote:
> Jean H. writes:
>
>> What is being displayed is a bit different than what is being stored..
>> IDL> print, 470.0 - (4.70*100)
>> 3.05176e-005
>
> I guess I would argue that what is being displayed is EXACTLY
> what is being stored:
Is it?
I always wondered if the garbage after the 16th digit does
correspond indeed to the exact decimal representation of the
double stored in memory in binary format, or is just a meaningless
side product of the binary to decimal conversion...
IDL> print,!DPi,format='(f68.64)'
3.1415926535897931159979634685441851615905761718750000000000 000000
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Ciao,
Paolo
>
> IDL> print, 470, format='(f18.14)'
> 470.00000000000000
> IDL> print, 4.70*100, format='(f18.14)'
> 469.99996948242187
> IDL> print, 470.00000000000000D - 469.99996948242187D, format='(f18.14)'
> 0.00003051757813
>
> Cheers,
>
> David
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
