| Re: bizarre number transformation [message #31487 is a reply to message #31485] |
Thu, 25 July 2002 05:11   |
wilms
Messages: 5
Registered: January 2001
|
Junior Member |
|
|
In article <b98a2b54.0207250346.381336bf@posting.google.com>,
merle <merlecorp@hotmail.com> wrote:
>
> The problem is that the number 443496.984 is being turned into the
> number 443496.969 from basic assignments using Float() or Double(),
> despite the fact that even floats should easily be able to handle a
> number this large (floats can handle "�10^38, with approximately six
> or seven decimal places of significance").
>
> Since I knew that I had successfully read in numbers much greater than
> 443496.984 in the past, I created temp.dat with just the number
> 443496.984 in it, and read this into a variable, x3. If x3 is cast as
> a float, it doesn't work, i.e. the number is 443496.969. But, if x3
> is cast as a double, then it contains the correct value. Why isn't a
> float sufficient (443496.984 << 10^38 and contains only 3 decimal
> places)? And, why doesn't x2=Double(443496.984) produce the correct
> result?
There seems to be a misconception on what a "decimal place" is. In your
example, your number has 9 decimal places, not three.
x2=double(443496.984) casts a single precision variable to double. If
you want to declare your number as a double, write
x2=443496.984D0
Cheers,
Joern
>
> Here's the code & the output:
>
> ; ------------------------------------------------------------ --------------
> PRO huh
>
> ; -- print with single precision -> NFG
> X1 = Float(443496.984)
> Print, X1, Format='("X1 = ", (F10.3))'
>
> ; -- print with double precision -> NFG
> X2 = Double(443496.984)
> Print, X2, Format='("X2 = ", (F10.3))'
>
>
> ; -- read the number from a file & print it out -> works fine
> OpenR, lun, 'temp.dat', /Get_Lun, ERROR = err
>
> ; -- Note: X3 must be cast as a double or else the number becomes
> 443496.969
> X3 = Double(0)
>
> ReadF, lun, X3
> Print, X3, Format='("X3 = ", (F10.3))'
>
> Free_Lun, lun
>
> END
>
>
> IDL> .COMPILE "D:\IDL\workdir\huh.pro"
> % Compiled module: HUH.
> IDL> huh
> X1 = 443496.969
> X2 = 443496.969
> X3 = 443496.984
>
> ; ------------------------------------------------------------ --------------
>
> Could someone please explain this to me? Or at least duplicate this
> error so that I don't think that I'm going crazy?
>
> thanks,
> merle
--
Dr. Joern Wilms Linsenbergstr. 33
Univ. Tuebingen, Institute for Astronomy D-72074 Tuebingen
Sand 1 (phone) +49 7071 29-76128
D-72076 Tuebingen, Germany wilms@astro.uni-tuebingen.de
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
