| Re: bizarre number transformation [message #31488 is a reply to message #31487] |
Thu, 25 July 2002 05:25   |
Don J Lindler
Messages: 19
Registered: April 2001
|
Junior Member |
|
|
>
> The problem is that the number 443496.984 is being turned into the
> number 443496.969 from basic assignments using Float() or Double(),
> despite the fact that even floats should easily be able to handle a
> number this large (floats can handle "�10^38, with approximately six
> or seven decimal places of significance").
>
When you count the seven decimal places of significance, you must count the
all of the digits, not just the ones to the right of the decimal point.
443496 and 9 are the seven digits of significance.
> Since I knew that I had successfully read in numbers much greater than
> 443496.984 in the past, I created temp.dat with just the number
> 443496.984 in it, and read this into a variable, x3. If x3 is cast as
> a float, it doesn't work, i.e. the number is 443496.969. But, if x3
> is cast as a double, then it contains the correct value. Why isn't a
> float sufficient (443496.984 << 10^38 and contains only 3 decimal
> places)? And, why doesn't x2=Double(443496.984) produce the correct
> result?
>
When you execute x2=Double(443496.984), IDL recongnizes 443496.984 as a
single precision floating point constant. It is placed into a single
precision temporary variable prior to conversion to double. The precision
is
already lost before the conversion. What you want is a double precision
constant:
x2 = 443496.984D0
Don
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
