comp.lang.idl-pvwave archive: archive » Re: Specify the degree of accuracy of a floating point number

Home » Public Forums » archive » Re: Specify the degree of accuracy of a floating point number

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Specify the degree of accuracy of a floating point number [message #52597 is a reply to message #52596]

Thu, 22 February 2007 10:28

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

Brian Larsen writes:

> Just to add more to the mix here:
>
> I use the solarsoft routine round_off.pro (I will put it inline as its
> short) to do this, seems to work well and its already written, which I
> always like.

To get this to work (I don't have the DATATYPE function), I changed
this case statement:

> case datatype(num) of
> 'BYT': x = byte(x)
> 'INT': x = fix(x)
> 'LON': x = long(x)
> 'DOU': x = double(x)
> 'FLO': x = float(x)
> endcase

To this:

case Size(num, /TName) of
'BYTE': x = byte(x)
'INT': x = fix(x)
'LONG': x = long(x)
'DOUBLE': x = double(x)
'FLOAT': x = float(x)
endcase

A pretty harmless change, it seems to me. Then I tried the
program:

IDL> a = 432.49584738273845D
IDL> print, round_off(a, 0.0000001)
2.9991178

Huh!? What did I do wrong?

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

[Message index]

		Re: Specify the degree of accuracy of a floating point number By: tochaf on Fri, 23 February 2007 02:01
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 13:18
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 12:43
		Re: Specify the degree of accuracy of a floating point number By: loknath on Thu, 22 February 2007 10:57
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 10:28
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 09:58
		Re: Specify the degree of accuracy of a floating point number By: Allan Whiteford on Thu, 22 February 2007 05:05
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Wed, 21 February 2007 22:22
		Re: Specify the degree of accuracy of a floating point number By: Kenneth Bowman on Fri, 23 February 2007 08:43

Previous Topic:	Re: Displaying slices using same scaling
Next Topic:	READ issue

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Tue Oct 14 14:10:39 PDT 2025

Total time taken to generate the page: 2.63962 seconds