| Specify the degree of accuracy of a floating point number [message #52607] |
Wed, 21 February 2007 20:48  |
loknath
Messages: 11
Registered: May 2006
|
Junior Member |
|
|
Hi
Is there a way to convert a single precision or double precision
floating point number to an accuracy of specified decimal places? In
other words, if I want to write 0.268954 or 233.256 to an accuracy of
2 decimal places, i.e. 0.27 and 233.26, is there a way to do it?
Thanks
|
|
|
|
| Re: Specify the degree of accuracy of a floating point number [message #52726 is a reply to message #52607] |
Sat, 24 February 2007 07:12  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Kenneth Bowman writes:
> This is a round-off error issue. The function works by multiplying the
> input by a factor, rounding to a LONG, and then dividing by the same factor
> to convert back to a floating-point type.
>
> When you are trying to preserve a lot of precision (digits), you get round-off
> issues with the LONG. This can be avoided (in most cases), by rounding to
> a LONG64.
Ah, thanks for this. I was using the L64 keyword for ROUND
in NUMBER_FORMATTER, but I was doing some of the other math
with FLOAT and LONG. By changing these to DOUBLE and LONG64
I've managed to make NUMBER_FORMATTER work properly with
all the perverse cases I've tried.
I've put an updated version here:
http://www.dfanning.com/programs/number_formatter.pro
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
