| Re: 0=1 (Double precision/Long64) [message #69897] |
Thu, 25 February 2010 14:30 |
rogass
Messages: 200
Registered: April 2008
|
Senior Member |
|
|
On 25 Feb., 17:56, wlandsman <wlands...@gmail.com> wrote:
> I had a program recently fail because I did not realize that adding 1
> to a number does not necessarily change its value ;-)
>
> IDL> a = 4611686018427387947
> IDL> b = double(a)
> IDL> help,a,b
> A LONG64 = 4611686018427387947
> B DOUBLE = 4.6116860e+18
> IDL> print,a EQ b
> 1
> IDL> print,a+1
> 4611686018427387948
> IDL> print,(a+1) EQ b
> 1
>
> So b is equal to both a and a+1. My guess is that the values are
> getting converted to double precision prior to the equality test.
> But the LONG64 variable has more precision than a double precision
> variable, and that precision is lost during the conversion.
>
> I'm not sure that there a good general solution for comparing between
> different data types. But one needs to be careful when comparing
> LONG64 and double variables.
>
> --Wayne
Oh yes,
its like my *problem with correlate. I compared the computed
coefficient by r le 1 and due to different precisions sometimes the
correlation coefficient was virtually larger than 1 ;)
Cheers
CR
|
|
|
|
| Re: 0=1 (Double precision/Long64) [message #69899 is a reply to message #69897] |
Thu, 25 February 2010 12:14  |
wlandsman
Messages: 743
Registered: June 2000
|
Senior Member |
|
|
On Feb 25, 1:08 pm, pp <pp.pente...@gmail.com> wrote:
>
> I think that converting integer types to floating point types is the
> usual way languages deal with operations that mix them, so this is not
> an IDL specific issue.
Ideally, I think one would want two numeric variables, a and b, to be
considered equal if
double(a) = double(b) *and* long64(a) = long64(b)
since the double variable is higher precision in allowing fractional
values, but the long64 variable is higher precision in preserving all
digits of very large integers. But I agree that the IDL approach is
standard among languages that allow comparison of data of different
types. --Wayne
|
|
|
|
| Re: 0=1 (Double precision/Long64) [message #69903 is a reply to message #69899] |
Thu, 25 February 2010 10:08 |
penteado
Messages: 866
Registered: February 2018
|
Senior Member
Administrator |
|
|
On Feb 25, 1:56 pm, wlandsman <wlands...@gmail.com> wrote:
> So b is equal to both a and a+1. My guess is that the values are
> getting converted to double precision prior to the equality test.
> But the LONG64 variable has more precision than a double precision
> variable, and that precision is lost during the conversion.
>
> I'm not sure that there a good general solution for comparing between
> different data types. But one needs to be careful when comparing
> LONG64 and double variables.
I think that converting integer types to floating point types is the
usual way languages deal with operations that mix them, so this is not
an IDL specific issue. Probably because it happens more often that the
floating number is not an integer, and the integer is small enough to
be represented exactly in the floating type.
Note that long64(b) is not equal to a, because double types are not
precise to 1 part in 19. Double precision is only good to about 15
digits. For a number of that size in a double, only additions of the
order of 1000 would change the value of b.
For that number to fit in a floating type you would need a quadruple
precision type (128 bits), which gets to 34 digits. But IDL does not
currently have such a type.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
