comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » double precision?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Double precision [message #65795 is a reply to message #49718]

Wed, 18 March 2009 22:16

plim.dreaming
Messages: 22
Registered: February 2008

Junior Member

Later in the program I calculate the separation between points (x1,y1)
(x2,y2)
And for some of those points the program says that the pairs are the
same. But they are only the same if they are rounded off, the
difference often only shows up in the last 2 decimal places.
ya, i read that link, most of it at least.
One other thing is: lets say the print out is the issue; a case like
you pointed out above, then why is it that if I do:
b=string(num)
print,b will give me the rounded off number?

On Mar 18, 9:32 pm, David Fanning <n...@dfanning.com> wrote:
> plim.dream...@gmail.com writes:
>> Ok, this is probably dumb but I need help with it.
>> So i'm reading in some data which was in string format, several
>> columns of data, and the data in one of the columns is getting
>> truncated, rounded off. I want more significant figures.
>> I have:
>> pixmag = reform(double(dbwork[1,*]))
>> dbwork is the original data, it has outputs such as 100.261787
>> pixmag, however, rounds that off to 100.26179 which is causing errors.
>> So how do I bump this up to 14 decimal places? Where do I put the
>> "D"??
>
> What evidence do you have that the numbers are being
> "rounded off"?
>
> IDL> str = '6.1234567890'
> IDL> num = double(str)
> IDL> print, num, format='(F12.10)'
> 6.1234567890
>
> However,
>
> IDL> print, num
> 6.1234568
>
> Is the default format of the PRINT command your problem?
>
> This is always a useful article to read:
>
> http://www.dfanning.com/math_tips/sky_is_falling.html
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		double precision? By: adisn123 on Thu, 10 August 2006 13:18
		Re: Double precision By: David Fanning on Thu, 19 March 2009 21:28
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 21:08
		Re: Double precision By: David Fanning on Thu, 19 March 2009 12:21
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 12:02
		Re: Double precision By: Jean H. on Thu, 19 March 2009 05:55
		Re: Double precision By: David Fanning on Thu, 19 March 2009 05:34
		Re: Double precision By: Allan Whiteford on Thu, 19 March 2009 03:51
		Re: Double precision By: Chris[6] on Thu, 19 March 2009 01:18
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 00:36
		Re: Double precision By: David Fanning on Wed, 18 March 2009 22:31
		Re: Double precision By: plim.dreaming on Wed, 18 March 2009 22:16
		Re: Double precision By: David Fanning on Wed, 18 March 2009 21:32

Previous Topic:	Re: Mac Widget Limitation?
Next Topic:	Re: Span a graph across 2 spaces in a !P.MULTI environment?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 09:13:48 PDT 2025

Total time taken to generate the page: 0.00382 seconds