comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: rounding errors [message #24906 is a reply to message #24905]

Fri, 27 April 2001 09:07

Karl Schultz
Messages: 341
Registered: October 1999

Senior Member

You could try

d = 2.56989d

to use a double-precision literal.

Also,

b=double(2.56989)

causes a single-to-double conversion because the literal is
single-precision.

The IEEE mantissa (fractional part) of this single-precision literal is (in
hex): 48F228
The conversion of a single to double will make the mantissa something like
48F2280000000 because it just adds zeros to pad out the mantissa.
But the mantissa of 2.56989d is 48F227D028A1E.

Comparing:
48F2280000000 (single-precision literal converted to double)
48F227D028A1E (double precision literal stored as double)

The first number is slightly larger, which accounts for the extra non-zero
decimal digits you point out below. When you convert the 2.56989 literal to
a single-precision float, the last mantissa bit is rounded up to get as
close to the literal as possible. When you convert that single to a double,
the "too high" bit is simply carried over to the double precision mantissa
and the rest of the lesser-significant bits are zeroed out. So, the number
still seems "too high". But if you store a double-precision literal as a
double, all the bits in a double-precision mantissa are used and the result
is what you expect.

And '2.56989' is different from 2.56989 because the latter is an implied
single-precision floating point number. The former is a string which gets
converted directly to double-precision if you say double('2.56989').

"Dominic R. Scales" <Dominic.Scales@aerosensing.de> wrote in message
news:3AE9330C.29D059E9@aerosensing.de...
> HELP!
>
> What gives? Is there any numerical math guy/gal out there
> who can tell me how this happens? It seems to me, that
> the accuracy of the second/third cast ist WAY off.
>
> a=double('2.56989')
> b=double( 2.56989 )
> c=double(float('2.56989'))
> .
> print,a,b,c,format='(d)'
>
> 2.5698900000000000 <---- this is what i want to have
> 2.5698900222778320
> 2.5698900222778320
>

Report message to a moderator

[Message index]

		Re: rounding errors By: thompson on Fri, 27 April 2001 11:34
		Re: rounding errors By: Liam E. Gumley on Fri, 27 April 2001 10:01
		Re: rounding errors By: Karl Schultz on Fri, 27 April 2001 09:07
		Re: rounding errors By: Paul van Delst on Fri, 27 April 2001 08:47
		Re: rounding errors By: Craig Markwardt on Fri, 27 April 2001 08:52
		Re: rounding errors By: John-David T. Smith on Fri, 27 April 2001 08:39
		Re: rounding errors By: thompson on Fri, 27 April 2001 08:23
		Re: rounding errors By: Dominic R. Scales on Fri, 27 April 2001 07:41
		Re: rounding errors By: Randall Skelton on Fri, 27 April 2001 07:51
		Re: rounding errors By: Liam E. Gumley on Fri, 27 April 2001 07:21
		Re: rounding errors By: Craig Markwardt on Fri, 27 April 2001 06:44
		Re: rounding errors By: Randall Skelton on Fri, 27 April 2001 06:17
		Re: rounding errors By: Dominic R. Scales on Fri, 27 April 2001 04:38
		Re: rounding errors By: Alex Schuster on Fri, 27 April 2001 03:52
		Re: rounding errors By: James Kuyper on Wed, 02 May 2001 08:06
		Re: rounding errors By: James Kuyper on Wed, 02 May 2001 08:07

Previous Topic:	Re: AAARGH-2 !!! CALL_EXTERNAL and IDL5.2 (the solution)
Next Topic:	TLB-Widget focus

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 06:36:49 PDT 2025

Total time taken to generate the page: 1.52055 seconds