comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: rounding errors [message #24913 is a reply to message #24910]

Fri, 27 April 2001 08:23

thompson
Messages: 584
Registered: August 1991

Senior Member

"Dominic R. Scales" <Dominic.Scales@aerosensing.de> writes:

> HELP!

> What gives? Is there any numerical math guy/gal out there
> who can tell me how this happens? It seems to me, that
> the accuracy of the second/third cast ist WAY off.

> a=double('2.56989')
> b=double( 2.56989 )
> c=double(float('2.56989'))

> print,a,b,c,format='(d)'

> 2.5698900000000000 <---- this is what i want to have
> 2.5698900222778320
> 2.5698900222778320

The first question that comes into my mind is why don't you simply cast your
literal as double precision in the first place:

IDL> a = 2.56989d0
IDL> print,a,format='(d)'
2.5698900000000000

It's instructive to look at these numbers in the binary representation actually
used by the computer. The floating point number 2.56989 has the following
representations in hexadecimal and binary formats:

Hex: 40247914
Binary: 01000000001001000111100100010100
^^^^^^^^^^^^^^^^^^^^^^

when this is converted from floating point to double precision, as in your
examples b and c above, you get

Hex: 40048F2280000000
Binary: 010000000000010010001111001000101000000000000000000000000000 0000
^^^^^^^^^^^^^^^^^^^^^^

It's a little harder to tell in hex format, but the binary format makes it
plain that the mantissa part is exactly the same as in the original floating
point number, just shifted over a little (the exponent part is bigger in double
precision), and filled with zeroes, just as you would expect. The confusion
results from the distinction between decimal representation used by people and
the binary representation used by computers.

William Thompson

Report message to a moderator

[Message index]

		Re: rounding errors By: thompson on Fri, 27 April 2001 11:34
		Re: rounding errors By: Liam E. Gumley on Fri, 27 April 2001 10:01
		Re: rounding errors By: Karl Schultz on Fri, 27 April 2001 09:07
		Re: rounding errors By: Paul van Delst on Fri, 27 April 2001 08:47
		Re: rounding errors By: Craig Markwardt on Fri, 27 April 2001 08:52
		Re: rounding errors By: John-David T. Smith on Fri, 27 April 2001 08:39
		Re: rounding errors By: thompson on Fri, 27 April 2001 08:23
		Re: rounding errors By: Dominic R. Scales on Fri, 27 April 2001 07:41
		Re: rounding errors By: Randall Skelton on Fri, 27 April 2001 07:51
		Re: rounding errors By: Liam E. Gumley on Fri, 27 April 2001 07:21
		Re: rounding errors By: Craig Markwardt on Fri, 27 April 2001 06:44
		Re: rounding errors By: Randall Skelton on Fri, 27 April 2001 06:17
		Re: rounding errors By: Dominic R. Scales on Fri, 27 April 2001 04:38
		Re: rounding errors By: Alex Schuster on Fri, 27 April 2001 03:52
		Re: rounding errors By: James Kuyper on Wed, 02 May 2001 08:06
		Re: rounding errors By: James Kuyper on Wed, 02 May 2001 08:07

Previous Topic:	Re: AAARGH-2 !!! CALL_EXTERNAL and IDL5.2 (the solution)
Next Topic:	TLB-Widget focus

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 15:28:46 PDT 2025

Total time taken to generate the page: 0.32318 seconds