comp.lang.idl-pvwave archive: archive » Re: Specify the degree of accuracy of a floating point number

Home » Public Forums » archive » Re: Specify the degree of accuracy of a floating point number

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Specify the degree of accuracy of a floating point number [message #52595 is a reply to message #52594]

Thu, 22 February 2007 12:43

Brian Larsen
Messages: 270
Registered: June 2006

Senior Member

David,

hmmm...

>
> IDL> a = 432.49584738273845D
> IDL> print, round_off(a, 0.0000001)
> 2.9991178
>
> Huh!? What did I do wrong?

This is what it does for me (and I do have datatype)

IDL> a = 432.49584738273845D
IDL> print, round_off(a, 0.0000001)
% Compiled module: ROUND_OFF.
-214.74836
% Program caused arithmetic error: Floating illegal operand
IDL> print, round_off(a, 0.0000001d)
-214.74836
% Program caused arithmetic error: Floating illegal operand
IDL> help, a
A DOUBLE = 432.49585
IDL> print, round_off(a, 0.0001)
432.49580

I think the answer is in the warning (% Program caused arithmetic
error: Floating illegal operand)
look at this chunk of the code:
num = abs(num)
if acc lt 1.0 then begin
factor = round(1.0/acc)
x = long((num+0.5*acc)*factor)/double(factor)
endif else begin
x = long((long((num+acc/2.)/acc))*acc)
endelse
;; and add in a help statement
HELP, x, acc

I then get:
IDL> a = 432.49584738273845D
IDL> print, round_off(a, 0.0000001)
X DOUBLE = -214.74836
ACC FLOAT = 1.00000e-07
-214.74836
% Program caused arithmetic error: Floating illegal operand

Is it something to do with dividing a long by a double??
x = long((num+0.5*acc)*factor)/double(factor)

Or something to do with?
http://www.dfanning.com/math_tips/sky_is_falling.html

Odd that it works with smaller number of digits...
IDL> print, round_off(a, 0.000001)
X DOUBLE = 432.49585
ACC FLOAT = 1.00000e-06
432.49585
IDL> print, round_off(a, 0.0000001)
X DOUBLE = -214.74836
ACC FLOAT = 1.00000e-07
-214.74836

I bet you are in a better position to understand this than I am
however...

Brian

------------------------------------------------------------ ---------
Brian A. Larsen
Dept. of Physics
Space Science and Engineering Lab (SSEL)
Montana State University - Bozeman
Bozeman, MT 59717

Report message to a moderator

[Message index]

		Re: Specify the degree of accuracy of a floating point number By: tochaf on Fri, 23 February 2007 02:01
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 13:18
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 12:43
		Re: Specify the degree of accuracy of a floating point number By: loknath on Thu, 22 February 2007 10:57
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 10:28
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 09:58
		Re: Specify the degree of accuracy of a floating point number By: Allan Whiteford on Thu, 22 February 2007 05:05
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Wed, 21 February 2007 22:22
		Re: Specify the degree of accuracy of a floating point number By: Kenneth Bowman on Fri, 23 February 2007 08:43

Previous Topic:	Re: Displaying slices using same scaling
Next Topic:	READ issue

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Tue Apr 28 03:22:49 PDT 2026

Total time taken to generate the page: 1.20336 seconds