comp.lang.idl-pvwave archive: archive » Re: 0=1 (Double precision/Long64)

Home » Public Forums » archive » Re: 0=1 (Double precision/Long64)

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: 0=1 (Double precision/Long64) [message #69897]

Thu, 25 February 2010 14:30

rogass
Messages: 200
Registered: April 2008

Senior Member

On 25 Feb., 17:56, wlandsman <wlands...@gmail.com> wrote:
> I had a program recently fail because I did not realize that adding 1
> to a number does not necessarily change its value ;-)
>
> IDL> a = 4611686018427387947
> IDL> b = double(a)
> IDL> help,a,b
> A LONG64 = 4611686018427387947
> B DOUBLE = 4.6116860e+18
> IDL> print,a EQ b
> 1
> IDL> print,a+1
> 4611686018427387948
> IDL> print,(a+1) EQ b
> 1
>
> So b is equal to both a and a+1. My guess is that the values are
> getting converted to double precision prior to the equality test.
> But the LONG64 variable has more precision than a double precision
> variable, and that precision is lost during the conversion.
>
> I'm not sure that there a good general solution for comparing between
> different data types. But one needs to be careful when comparing
> LONG64 and double variables.
>
> --Wayne

Oh yes,
its like my *problem with correlate. I compared the computed
coefficient by r le 1 and due to different precisions sometimes the
correlation coefficient was virtually larger than 1 ;)

Cheers

CR

Report message to a moderator

[Message index]

		Re: 0=1 (Double precision/Long64) By: rogass on Thu, 25 February 2010 14:30
		Re: 0=1 (Double precision/Long64) By: wlandsman on Thu, 25 February 2010 12:14
		Re: 0=1 (Double precision/Long64) By: penteado on Thu, 25 February 2010 10:08

Previous Topic:	Print 2 arrays side by side in one file
Next Topic:	Re: MTPR error when I try to apply a mask

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 08:16:44 PDT 2025

Total time taken to generate the page: 0.88091 seconds