comp.lang.idl-pvwave archive: archive » Re: Precision Problem

Home » Public Forums » archive » Re: Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Precision Problem [message #49393]

Mon, 24 July 2006 12:34

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

Paul Van Delst wrote:
> You're correct. If you want more than 6/7 decimal place precision, you

I really should have said if you want more than 6/7 meaningful significant figures...

> need double (at least, for it to be meaningful):
>
> IDL> x=10000000.0
> IDL> print, x, format='(f12.2)'
> 10000000.00
> IDL> x=x+0.1
> IDL> print, x, format='(f12.2)'
> 10000000.00
>
> IDL> x=10000000.0d
> IDL> print, x, format='(f12.2)'
> 10000000.00
> IDL> x=x+0.1d
> IDL> print, x, format='(f12.2)'
> 10000000.10
>
>
> paulv
>

--
Paul van Delst Ride lots.
CIMSS @ NOAA/NCEP/EMC Eddy Merckx
Ph: (301)763-8000 x7748
Fax:(301)763-8545

Report message to a moderator

[Message index]

		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:34
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:33
		Re: Precision Problem By: Bruce Bowler on Mon, 24 July 2006 12:19
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:00
		Re: Precision Problem By: DMac on Mon, 24 July 2006 14:05

Previous Topic:	Close window command?
Next Topic:	Re: problem loading color palette

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Nov 30 04:55:24 PST 2025

Total time taken to generate the page: 0.87998 seconds