comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Precision Problem

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: precision problem [message #67663 is a reply to message #49403]

Tue, 18 August 2009 05:58

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

skymaxwell@gmail.com writes:

> I have problem with precision.
> I typing
>
> IDL> b=13.37943889D
>
> and then i typing
>
> IDL> help,b
> B DOUBLE = 13.379439
>
> how get my 13.37943889 for calculus ?
> If IDL rounding value, i've got worst results
> Even for test example
>
> How avoid this problem ?

More reading, naturally.

http://www.dfanning.com/math_tips/sky_is_falling.html
http://www.dfanning.com/math_tips/double.html
http://www.dfanning.com/misc_tips/dbl_to_str.html

Cheers,

David

--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		Precision Problem By: DMac on Mon, 24 July 2006 09:51
		Re: Precision Problem By: Mark Hadfield on Mon, 24 July 2006 15:10
		Re: precision problem By: David Fanning on Tue, 18 August 2009 05:58

Previous Topic:	precision problem
Next Topic:	Read BUFR Meteorological Data in IDL?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 13:45:35 PDT 2025

Total time taken to generate the page: 2.23849 seconds