comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: rounding [message #32002 is a reply to message #24924]

Wed, 04 September 2002 07:52

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

Amar Nayegandhi (anayegan@csee.usf.edu) writes:

> IDL seems to round off large decimal numbers. For e.g.,
> IDL> x = 284.766117
> IDL> print, format='(I11)',x*1000000L
> 284766112
>
> IDL> x = 284.766119
> IDL> print, format='(I11)',x*1000000L
> 284766112
>
> Is there any way around this?

Here is an article that might shed some light on this:

http://www.dfanning.com/math_tips/sky_is_falling.html

Cheers,

David

--
David W. Fanning, Ph.D.
Fanning Software Consulting, Inc.
Phone: 970-221-0438, E-mail: david@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155

Report message to a moderator

[Message index]

		rounding errors By: Dominic R. Scales on Fri, 27 April 2001 01:51
		Re: rounding errors By: Jaco van Gorkom on Fri, 04 May 2001 06:17
		Re: rounding errors By: Paul van Delst on Wed, 02 May 2001 09:46
		Re: rounding errors By: Dominic R. Scales on Wed, 02 May 2001 00:44
		Re: rounding By: thompson on Wed, 04 September 2002 07:59
		Re: rounding By: David Fanning on Wed, 04 September 2002 07:52
		Re: rounding By: Craig Markwardt on Wed, 04 September 2002 07:45
		Re: rounding By: Amar Nayegandhi on Wed, 04 September 2002 07:40

Previous Topic:	rounding
Next Topic:	Object Graphics Book.

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 07:24:08 PDT 2025

Total time taken to generate the page: 0.96292 seconds