comp.lang.idl-pvwave archive: archive » Re: machine precision

Home » Public Forums » archive » Re: machine precision

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: machine precision [message #66497 is a reply to message #66496]

Tue, 19 May 2009 08:16

jeffnettles4870
Messages: 111
Registered: October 2006

Senior Member

On May 18, 9:44 pm, James Kuyper <jameskuy...@verizon.net> wrote:
> Wox wrote:
>> On Mon, 18 May 2009 13:33:48 GMT, James Kuyper
>> <jameskuy...@verizon.net> wrote:
>
>>> At a minimum, you should use pres*(f1^2+f2^2)^0.5 instead of pres for
>>> the following comparison. eps gives you the relative precision, not the
>>> absolute precision; it needs to be scaled by the numbers you're working
>>> with.
>
>> Now this I didn't realize! So it's the precision of the mantisse or
>> something?
>
> Yes. If you don't understand why, I recommend re-reading the "The Sky is
> Falling!" website that David referred you to, and in particular the
> paper it refers to titled "What Every Computer Scientist Should Know
> about Floating Point Arithmetic".

I think I have the beginnings of an understanding of this problem
after having seen the topic pop up so much here in the newsgroup and
after having read David's article several times. But now i'm
wondering how other languages handle this situation? I've tried two
other languages now, perl and VB, and they both seem to not give the
"correct-but-not-expected" results that trip so many people up. For
example, this snippet of perl code:

$i = int(4.70*100);
print "$i\n";

prints 470 whereas the equivalent IDL code: print, fix(4.70*100)
prints 469.

Does anyone have any idea how other languages deal with the "sky is
falling" problem?

Jeff

Report message to a moderator

[Message index]

		Re: machine precision By: Wout De Nolf on Wed, 20 May 2009 06:08
		Re: machine precision By: Carsten Lechte on Wed, 20 May 2009 03:44
		Re: machine precision By: jameskuyper on Wed, 20 May 2009 03:28
		Re: machine precision By: Carsten Lechte on Wed, 20 May 2009 01:46
		Re: machine precision By: David Fanning on Tue, 19 May 2009 20:01
		Re: machine precision By: jameskuyper on Tue, 19 May 2009 18:41
		Re: machine precision By: Kenneth P. Bowman on Tue, 19 May 2009 11:15
		Re: machine precision By: jeffnettles4870 on Tue, 19 May 2009 08:16
		Re: machine precision By: jameskuyper on Mon, 18 May 2009 18:44
		Re: machine precision By: Wout De Nolf on Mon, 18 May 2009 07:02
		Re: machine precision By: Wout De Nolf on Mon, 18 May 2009 06:55
		Re: machine precision By: jameskuyper on Mon, 18 May 2009 06:33
		Re: machine precision By: David Fanning on Mon, 18 May 2009 06:30
		Re: machine precision By: Wout De Nolf on Mon, 18 May 2009 06:04
		Re: machine precision By: David Fanning on Mon, 18 May 2009 04:53

Previous Topic:	Re: Simultaneous fitting in IDL
Next Topic:	Simultaneous fitting in IDL

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 22:17:56 PDT 2025

Total time taken to generate the page: 1.12018 seconds