comp.lang.idl-pvwave archive: archive » Re: Specify the degree of accuracy of a floating point number

Home » Public Forums » archive » Re: Specify the degree of accuracy of a floating point number

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Specify the degree of accuracy of a floating point number [message #52596 is a reply to message #52595]

Thu, 22 February 2007 10:57

loknath
Messages: 11
Registered: May 2006

Junior Member

On Feb 22, 11:28 am, David Fanning <n...@dfanning.com> wrote:
> Brian Larsen writes:
>> Just to add more to the mix here:
>
>> I use the solarsoft routine round_off.pro (I will put it inline as its
>> short) to do this, seems to work well and its already written, which I
>> always like.
>
> To get this to work (I don't have the DATATYPE function), I changed
> this case statement:
>
>> case datatype(num) of
>> 'BYT': x = byte(x)
>> 'INT': x = fix(x)
>> 'LON': x = long(x)
>> 'DOU': x = double(x)
>> 'FLO': x = float(x)
>> endcase
>
> To this:
>
> case Size(num, /TName) of
> 'BYTE': x = byte(x)
> 'INT': x = fix(x)
> 'LONG': x = long(x)
> 'DOUBLE': x = double(x)
> 'FLOAT': x = float(x)
> endcase
>
> A pretty harmless change, it seems to me. Then I tried the
> program:
>
> IDL> a = 432.49584738273845D
> IDL> print, round_off(a, 0.0000001)
> 2.9991178
>
> Huh!? What did I do wrong?
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")

thank you,
all of you for the great suggestions.
Loknath

Report message to a moderator

[Message index]

		Re: Specify the degree of accuracy of a floating point number By: tochaf on Fri, 23 February 2007 02:01
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 13:18
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 12:43
		Re: Specify the degree of accuracy of a floating point number By: loknath on Thu, 22 February 2007 10:57
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Thu, 22 February 2007 10:28
		Re: Specify the degree of accuracy of a floating point number By: Brian Larsen on Thu, 22 February 2007 09:58
		Re: Specify the degree of accuracy of a floating point number By: Allan Whiteford on Thu, 22 February 2007 05:05
		Re: Specify the degree of accuracy of a floating point number By: David Fanning on Wed, 21 February 2007 22:22
		Re: Specify the degree of accuracy of a floating point number By: Kenneth Bowman on Fri, 23 February 2007 08:43

Previous Topic:	Re: Displaying slices using same scaling
Next Topic:	READ issue

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 23:23:51 PDT 2025

Total time taken to generate the page: 0.79690 seconds