| Re: DOUBLE trouble [message #14555] |
Tue, 09 March 1999 00:00 |
Axel vom Endt
Messages: 9
Registered: March 1999
|
Junior Member |
|
|
Hi Jacco,
there are two things going on:
IDL like most programming languages uses hardware floating point
numbers. On all machines I've ever used that means using binary
fractions. 3.0 becomes 11.0 (which then will be stored as .11 in the
mantissa). 3.1 becomes a periodic binary fraction 11.0001100110011 which
cannot be stored without roundoff error. I hope I got the numbers right,
but this is explained in any textbook on numerical methods.
The second issue is printing in IDL. Try using an explicit FORMAT
statement
IDL> print, 3.1d0, format='(f60.55)'
3.1000000000000000888178419700125232338905334472656250000
IDL> print, 3.1e0, format='(f60.55)'
3.0999999046325683593750000000000000000000000000000000000
and you'll see that neither double nor single precision will be exactly
3.1
Axel
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
