comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » double precision?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Double precision [message #65789 is a reply to message #49718]

Thu, 19 March 2009 05:55

Jean H.
Messages: 472
Registered: July 2006

Senior Member

plim.dreaming@gmail.com wrote:
> Later in the program I calculate the separation between points (x1,y1)
> (x2,y2)
> And for some of those points the program says that the pairs are the
> same. But they are only the same if they are rounded off, the
> difference often only shows up in the last 2 decimal places.
> ya, i read that link, most of it at least.
> One other thing is: lets say the print out is the issue; a case like
> you pointed out above, then why is it that if I do:
> b=string(num)
> print,b will give me the rounded off number?

as David pointed it out, be careful on how you compare your numbers...
a = 0.1234567890123456d
b = 0.1234567890123456d
print, a eq b
1

b = 0.1234567890123456d + 0.1 - 0.1
print, a eq b
0

Also, I suggest you to print your number value a few times (after
reading, after allocating to pixmag, before comparing it), with the
proper format ...
IDL> c = 100.489418d
IDL> print, c
100.48942
IDL> print, c, format = '(F15.10)'
100.4894180000

Jean

Report message to a moderator

[Message index]

		double precision? By: adisn123 on Thu, 10 August 2006 13:18
		Re: Double precision By: David Fanning on Thu, 19 March 2009 21:28
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 21:08
		Re: Double precision By: David Fanning on Thu, 19 March 2009 12:21
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 12:02
		Re: Double precision By: Jean H. on Thu, 19 March 2009 05:55
		Re: Double precision By: David Fanning on Thu, 19 March 2009 05:34
		Re: Double precision By: Allan Whiteford on Thu, 19 March 2009 03:51
		Re: Double precision By: Chris[6] on Thu, 19 March 2009 01:18
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 00:36
		Re: Double precision By: David Fanning on Wed, 18 March 2009 22:31
		Re: Double precision By: plim.dreaming on Wed, 18 March 2009 22:16
		Re: Double precision By: David Fanning on Wed, 18 March 2009 21:32

Previous Topic:	Re: Mac Widget Limitation?
Next Topic:	Re: Span a graph across 2 spaces in a !P.MULTI environment?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 19:23:25 PDT 2025

Total time taken to generate the page: 0.00396 seconds