comp.lang.idl-pvwave archive: archive » Re: Precision Problem

Home » Public Forums » archive » Re: Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Precision Problem [message #49486 is a reply to message #49393]

Mon, 24 July 2006 14:05

DMac
Messages: 6
Registered: May 2006

Junior Member

Thanks a bunch guys. That was the problem i.e. I should use 1.000d-002
rather than the single precision literal 1.000e-002.

Solved all my problems.

I really appreciate the prompt and accurate replies.

Derek M.

Paul Van Delst wrote:
> Paul Van Delst wrote:
>> You're correct. If you want more than 6/7 decimal place precision, you
>
> I really should have said if you want more than 6/7 meaningful significant figures...
>
>
>> need double (at least, for it to be meaningful):
>>
>> IDL> x=10000000.0
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>> IDL> x=x+0.1
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>>
>> IDL> x=10000000.0d
>> IDL> print, x, format='(f12.2)'
>> 10000000.00
>> IDL> x=x+0.1d
>> IDL> print, x, format='(f12.2)'
>> 10000000.10
>>
>>
>> paulv
>>
>
>
> --
> Paul van Delst Ride lots.
> CIMSS @ NOAA/NCEP/EMC Eddy Merckx
> Ph: (301)763-8000 x7748
> Fax:(301)763-8545

Report message to a moderator

[Message index]

		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:34
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:33
		Re: Precision Problem By: Bruce Bowler on Mon, 24 July 2006 12:19
		Re: Precision Problem By: Paul Van Delst[1] on Mon, 24 July 2006 12:00
		Re: Precision Problem By: DMac on Mon, 24 July 2006 14:05

Previous Topic:	Close window command?
Next Topic:	Re: problem loading color palette

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 04:51:14 PDT 2025

Total time taken to generate the page: 1.19962 seconds