comp.lang.idl-pvwave archive: archive » Specify the degree of accuracy of a floating point number

Home » Public Forums » archive » Specify the degree of accuracy of a floating point number

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Specify the degree of accuracy of a floating point number [message #52726 is a reply to message #52607]

Sat, 24 February 2007 07:12

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

Kenneth Bowman writes:

> This is a round-off error issue. The function works by multiplying the
> input by a factor, rounding to a LONG, and then dividing by the same factor
> to convert back to a floating-point type.
>
> When you are trying to preserve a lot of precision (digits), you get round-off
> issues with the LONG. This can be avoided (in most cases), by rounding to
> a LONG64.

Ah, thanks for this. I was using the L64 keyword for ROUND
in NUMBER_FORMATTER, but I was doing some of the other math
with FLOAT and LONG. By changing these to DOUBLE and LONG64
I've managed to make NUMBER_FORMATTER work properly with
all the perverse cases I've tried.

I've put an updated version here:

http://www.dfanning.com/programs/number_formatter.pro

Cheers,

David

--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		Specify the degree of accuracy of a floating point number By: loknath on Wed, 21 February 2007 20:48
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Sat, 24 February 2007 07:12

Previous Topic:	Re: TRIANGULATE. Finding contiguous cells efficiently?
Next Topic:	Fast way to calculate sum of discrete poisson distribution

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 02:26:10 PDT 2025

Total time taken to generate the page: 0.00814 seconds