comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » double precision?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Double precision [message #65796 is a reply to message #49718]

Wed, 18 March 2009 21:32

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

plim.dreaming@gmail.com writes:

> Ok, this is probably dumb but I need help with it.
> So i'm reading in some data which was in string format, several
> columns of data, and the data in one of the columns is getting
> truncated, rounded off. I want more significant figures.
> I have:
> pixmag = reform(double(dbwork[1,*]))
> dbwork is the original data, it has outputs such as 100.261787
> pixmag, however, rounds that off to 100.26179 which is causing errors.
> So how do I bump this up to 14 decimal places? Where do I put the
> "D"??

What evidence do you have that the numbers are being
"rounded off"?

IDL> str = '6.1234567890'
IDL> num = double(str)
IDL> print, num, format='(F12.10)'
6.1234567890

However,

IDL> print, num
6.1234568

Is the default format of the PRINT command your problem?

This is always a useful article to read:

http://www.dfanning.com/math_tips/sky_is_falling.html

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		double precision? By: adisn123 on Thu, 10 August 2006 13:18
		Re: Double precision By: David Fanning on Thu, 19 March 2009 21:28
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 21:08
		Re: Double precision By: David Fanning on Thu, 19 March 2009 12:21
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 12:02
		Re: Double precision By: Jean H. on Thu, 19 March 2009 05:55
		Re: Double precision By: David Fanning on Thu, 19 March 2009 05:34
		Re: Double precision By: Allan Whiteford on Thu, 19 March 2009 03:51
		Re: Double precision By: Chris[6] on Thu, 19 March 2009 01:18
		Re: Double precision By: plim.dreaming on Thu, 19 March 2009 00:36
		Re: Double precision By: David Fanning on Wed, 18 March 2009 22:31
		Re: Double precision By: plim.dreaming on Wed, 18 March 2009 22:16
		Re: Double precision By: David Fanning on Wed, 18 March 2009 21:32

Previous Topic:	Re: Mac Widget Limitation?
Next Topic:	Re: Span a graph across 2 spaces in a !P.MULTI environment?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 09:14:34 PDT 2025

Total time taken to generate the page: 0.01465 seconds