comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: rounding [message #32003 is a reply to message #24924]

Wed, 04 September 2002 07:45

Craig Markwardt
Messages: 1869
Registered: November 1996

Senior Member

Amar Nayegandhi <anayegan@csee.usf.edu> writes:
> Hi,
> IDL seems to round off large decimal numbers. For e.g.,
> IDL> x = 284.766117
> IDL> print, format='(I11)',x*1000000L
> 284766112
>
> IDL> x = 284.766119
> IDL> print, format='(I11)',x*1000000L
> 284766112
>
> Is there any way around this?

Yes, use double precision.

x = 284.766117D

Of course, that only gets you out to 17 digits of precision or so.
Normal FLOATs keep about 7 digits.

Craig

--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------

Report message to a moderator

[Message index]

		rounding errors By: Dominic R. Scales on Fri, 27 April 2001 01:51
		Re: rounding errors By: Jaco van Gorkom on Fri, 04 May 2001 06:17
		Re: rounding errors By: Paul van Delst on Wed, 02 May 2001 09:46
		Re: rounding errors By: Dominic R. Scales on Wed, 02 May 2001 00:44
		Re: rounding By: thompson on Wed, 04 September 2002 07:59
		Re: rounding By: David Fanning on Wed, 04 September 2002 07:52
		Re: rounding By: Craig Markwardt on Wed, 04 September 2002 07:45
		Re: rounding By: Amar Nayegandhi on Wed, 04 September 2002 07:40

Previous Topic:	rounding
Next Topic:	Object Graphics Book.

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 14:38:28 PDT 2025

Total time taken to generate the page: 0.48846 seconds