comp.lang.idl-pvwave archive: archive » Re: 0=1 (Double precision/Long64)

Home » Public Forums » archive » Re: 0=1 (Double precision/Long64)

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: 0=1 (Double precision/Long64) [message #69903 is a reply to message #69899]

Thu, 25 February 2010 10:08

penteado
Messages: 866
Registered: February 2018

Senior Member
Administrator

On Feb 25, 1:56 pm, wlandsman <wlands...@gmail.com> wrote:
> So b is equal to both a and a+1. My guess is that the values are
> getting converted to double precision prior to the equality test.
> But the LONG64 variable has more precision than a double precision
> variable, and that precision is lost during the conversion.
>
> I'm not sure that there a good general solution for comparing between
> different data types. But one needs to be careful when comparing
> LONG64 and double variables.

I think that converting integer types to floating point types is the
usual way languages deal with operations that mix them, so this is not
an IDL specific issue. Probably because it happens more often that the
floating number is not an integer, and the integer is small enough to
be represented exactly in the floating type.

Note that long64(b) is not equal to a, because double types are not
precise to 1 part in 19. Double precision is only good to about 15
digits. For a number of that size in a double, only additions of the
order of 1000 would change the value of b.

For that number to fit in a floating type you would need a quadruple
precision type (128 bits), which gets to 34 digits. But IDL does not
currently have such a type.

Report message to a moderator

[Message index]

		Re: 0=1 (Double precision/Long64) By: rogass on Thu, 25 February 2010 14:30
		Re: 0=1 (Double precision/Long64) By: wlandsman on Thu, 25 February 2010 12:14
		Re: 0=1 (Double precision/Long64) By: penteado on Thu, 25 February 2010 10:08

Previous Topic:	Print 2 arrays side by side in one file
Next Topic:	Re: MTPR error when I try to apply a mask

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Nov 30 15:08:35 PST 2025

Total time taken to generate the page: 1.35929 seconds