comp.lang.idl-pvwave archive: archive » 0=1 (Double precision/Long64)

Home » Public Forums » archive » 0=1 (Double precision/Long64)

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

0=1 (Double precision/Long64) [message #69904]

Thu, 25 February 2010 08:56

wlandsman
Messages: 743
Registered: June 2000

Senior Member

I had a program recently fail because I did not realize that adding 1
to a number does not necessarily change its value ;-)

IDL> a = 4611686018427387947
IDL> b = double(a)
IDL> help,a,b
A LONG64 = 4611686018427387947
B DOUBLE = 4.6116860e+18
IDL> print,a EQ b
1
IDL> print,a+1
4611686018427387948
IDL> print,(a+1) EQ b
1

So b is equal to both a and a+1. My guess is that the values are
getting converted to double precision prior to the equality test.
But the LONG64 variable has more precision than a double precision
variable, and that precision is lost during the conversion.

I'm not sure that there a good general solution for comparing between
different data types. But one needs to be careful when comparing
LONG64 and double variables.

--Wayne

Report message to a moderator

[Message index]

0=1 (Double precision/Long64)

By: wlandsman on Thu, 25 February 2010 08:56

Previous Topic:	Pri
Next Topic:	Grids on Minor Ticks.

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 11:25:13 PDT 2025

Total time taken to generate the page: 1.03996 seconds