comp.lang.idl-pvwave archive: archive » Re: Specify the degree of accuracy of a floating point number

Home » Public Forums » archive » Re: Specify the degree of accuracy of a floating point number

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Specify the degree of accuracy of a floating point number [message #52730 is a reply to message #52597]

Fri, 23 February 2007 08:43

Kenneth Bowman
Messages: 86
Registered: November 2006

Member

In article <MPG.20478eaa542e4575989e91@news.frii.com>,
David Fanning <news@dfanning.com> wrote:

> A pretty harmless change, it seems to me. Then I tried the
> program:
>
> IDL> a = 432.49584738273845D
> IDL> print, round_off(a, 0.0000001)
> 2.9991178
>
> Huh!? What did I do wrong?

This is a round-off error issue. The function works by multiplying the
input by a factor, rounding to a LONG, and then dividing by the same factor
to convert back to a floating-point type.

When you are trying to preserve a lot of precision (digits), you get round-off
issues with the LONG. This can be avoided (in most cases), by rounding to
a LONG64.

See examples below

IDL> a = 432.49584738273845D

This works for small precision
IDL> print, DOUBLE(ROUND(100.0D0*a))/100.0
432.50000

This fails for large precision
IDL> print, DOUBLE(ROUND(100000000.0D0*a))/100000000.0
21.474836
% Program caused arithmetic error: Floating illegal operand

But works if 64-bit integers are used
IDL> print, DOUBLE(ROUND(100000000.0D0*a, /L64))/100000000.0
432.49585

Cheers, Ken

Report message to a moderator

[Message index]

		Re: Specify the degree of accuracy of a floating point number By: tochaf on Fri, 23 February 2007 02:01
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 13:18
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 12:43
		Re: Specify the degree of accuracy of a floating point number By: loknath on Thu, 22 February 2007 10:57
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 10:28
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 09:58
		Re: Specify the degree of accuracy of a floating point number By: Allan Whiteford on Thu, 22 February 2007 05:05
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Wed, 21 February 2007 22:22
		Re: Specify the degree of accuracy of a floating point number By: Kenneth Bowman on Fri, 23 February 2007 08:43

Previous Topic:	Re: Displaying slices using same scaling
Next Topic:	READ issue

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 04:23:12 PDT 2025

Total time taken to generate the page: 1.17115 seconds