comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: rounding [message #32001 is a reply to message #24924]

Wed, 04 September 2002 07:59

thompson
Messages: 584
Registered: August 1991

Senior Member

Amar Nayegandhi <anayegan@csee.usf.edu> writes:

> Hi,
> IDL seems to round off large decimal numbers. For e.g.,
> IDL> x = 284.766117
> IDL> print, format='(I11)',x*1000000L
> 284766112

> IDL> x = 284.766119
> IDL> print, format='(I11)',x*1000000L
> 284766112

> Is there any way around this?

This isn't IDL, it's just the way binary computers work. To get around it, use
double precision instead of single precision.

IDL> x = 284.766117d0
IDL> print, format='(I11)',x*1000000L
284766117

Another way to look at what's going on is to print out the floating point
numbers to high precision.

IDL> x = 284.766117
IDL> print, format='(F22.11)',x
284.76611328125
IDL> x = 284.766117d0
IDL> print, format='(F22.11)',x
284.76611700000
IDL> print, format='(F27.16)',x
284.7661170000000100

It isn't that IDL rounds off the numbers, it's that some precision is lost in
converting back and forth between the decimal notation that people use and the
binary notation that computers use internally. Double precision numbers
have the same problem, as shown above, only not so bad.

William Thompson

Report message to a moderator

[Message index]

		rounding errors By: Dominic R. Scales on Fri, 27 April 2001 01:51
		Re: rounding errors By: Jaco van Gorkom on Fri, 04 May 2001 06:17
		Re: rounding errors By: Paul van Delst on Wed, 02 May 2001 09:46
		Re: rounding errors By: Dominic R. Scales on Wed, 02 May 2001 00:44
		Re: rounding By: thompson on Wed, 04 September 2002 07:59
		Re: rounding By: David Fanning on Wed, 04 September 2002 07:52
		Re: rounding By: Craig Markwardt on Wed, 04 September 2002 07:45
		Re: rounding By: Amar Nayegandhi on Wed, 04 September 2002 07:40

Previous Topic:	rounding
Next Topic:	Object Graphics Book.

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 05:42:29 PDT 2025

Total time taken to generate the page: 0.16175 seconds