comp.lang.idl-pvwave archive: archive » Re: HISTOGRAM and the Razor's Edge.

Home » Public Forums » archive » Re: HISTOGRAM and the Razor's Edge.

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: HISTOGRAM and the Razor's Edge. [message #35447 is a reply to message #35436]

Thu, 12 June 2003 08:03

meinel
Messages: 14
Registered: February 1994

Junior Member

Craig Markwardt wrote ...
> timrobishaw@yahoo.com (Tim Robishaw) writes:
> ...
>> Well, there ya go. It's a roundoff error problem that results from
>> trying to balance the values on a razor's edge... the subtraction and
>> division knock a few values off balance.
>
> Partial solution number four: work in powers of 2 instead of multiples
> of 0.05.

Isn't this in the FAQ somewhere?

The technical answer: Computers represent numbers as base 2; humans
represent numbers as base 10. Mapping numbers from base 10 to base 2
is exact, mapping numbers from base 10 to base 2 ain't.

5+5/10 = 5+1/2 exact correspondence

5+45/100 = 5+1/4+1/8+1/16+1/128+1/256+1/2048+... not exact, even in
DP

So, depending on the machine precision, 5.45 on the computer is either
slightly greater than or less than an exact representation of 5.45. On
top of that, neither is 0.05 exact on the computer, so your bin size
is also slightly different than what you are expecting.

Bottom line: if you want the results to be exact, think like a
machine.

Ed

PS. Let's just say, ...

Report message to a moderator

[Message index]

		Re: HISTOGRAM and the Razor's Edge. By: Paul Van Delst[1] on Fri, 13 June 2003 06:58
		Re: HISTOGRAM and the Razor's Edge. By: JD Smith on Thu, 12 June 2003 15:25
		Re: HISTOGRAM and the Razor's Edge. By: timrobishaw on Thu, 12 June 2003 14:54
		Re: HISTOGRAM and the Razor's Edge. By: meinel on Thu, 12 June 2003 08:03
		Re: HISTOGRAM and the Razor's Edge. By: Paul Van Delst[1] on Thu, 12 June 2003 07:40
		Re: HISTOGRAM and the Razor's Edge. By: David Fanning on Thu, 12 June 2003 07:30
		Re: HISTOGRAM and the Razor's Edge. By: James Kuyper on Thu, 12 June 2003 06:53
		Re: HISTOGRAM and the Razor's Edge. By: R.G. Stockwell on Thu, 12 June 2003 04:33
		Re: HISTOGRAM and the Razor's Edge. By: Craig Markwardt on Thu, 12 June 2003 00:04

Previous Topic:	how did this happen?
Next Topic:	Re: Redirect STDOUT

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Thu Oct 09 23:02:53 PDT 2025

Total time taken to generate the page: 1.67517 seconds