| bizarre number transformation [message #31489] |
Thu, 25 July 2002 04:46  |
merlecorp
Messages: 2
Registered: May 2002
|
Junior Member |
|
|
Hello,
I ran into a number transformation error yesterday that is still
confusing me this morning. At first I thought I was doing something
silly with String() & StrTrim(), but then I wrote a little program
(see huh.pro) with no conversions that still contained the problem.
FYI, I'm using IDL Version 5.5 Win32 (x86).
The problem is that the number 443496.984 is being turned into the
number 443496.969 from basic assignments using Float() or Double(),
despite the fact that even floats should easily be able to handle a
number this large (floats can handle "�10^38, with approximately six
or seven decimal places of significance").
Since I knew that I had successfully read in numbers much greater than
443496.984 in the past, I created temp.dat with just the number
443496.984 in it, and read this into a variable, x3. If x3 is cast as
a float, it doesn't work, i.e. the number is 443496.969. But, if x3
is cast as a double, then it contains the correct value. Why isn't a
float sufficient (443496.984 << 10^38 and contains only 3 decimal
places)? And, why doesn't x2=Double(443496.984) produce the correct
result?
Here's the code & the output:
; ------------------------------------------------------------ --------------
PRO huh
; -- print with single precision -> NFG
X1 = Float(443496.984)
Print, X1, Format='("X1 = ", (F10.3))'
; -- print with double precision -> NFG
X2 = Double(443496.984)
Print, X2, Format='("X2 = ", (F10.3))'
; -- read the number from a file & print it out -> works fine
OpenR, lun, 'temp.dat', /Get_Lun, ERROR = err
; -- Note: X3 must be cast as a double or else the number becomes
443496.969
X3 = Double(0)
ReadF, lun, X3
Print, X3, Format='("X3 = ", (F10.3))'
Free_Lun, lun
END
IDL> .COMPILE "D:\IDL\workdir\huh.pro"
% Compiled module: HUH.
IDL> huh
X1 = 443496.969
X2 = 443496.969
X3 = 443496.984
; ------------------------------------------------------------ --------------
Could someone please explain this to me? Or at least duplicate this
error so that I don't think that I'm going crazy?
thanks,
merle
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
