comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » rounding errors

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

rounding errors [message #24924]

Fri, 27 April 2001 01:51

Dominic R. Scales
Messages: 12
Registered: April 2001

Junior Member

HELP!

What gives? Is there any numerical math guy/gal out there
who can tell me how this happens? It seems to me, that
the accuracy of the second/third cast ist WAY off.

a=double('2.56989')
b=double( 2.56989 )
c=double(float('2.56989'))

print,a,b,c,format='(d)'

2.5698900000000000 <---- this is what i want to have
2.5698900222778320
2.5698900222778320

So, the question is: why can the cast from a string give a much
more accurate result than a cast from a literal constant (or a
float variable, for that matter)? Do I really have to cast with:

d=double(2.56989*1000000L)/1000000.
print,d,format='(d)'

2.5698900000000000

or even
d=double(string(2.56989))
print,d,format='(d)'

2.5698900000000000

Ah yes, and while I'm at it... Have you ever compared
5.2e-6, 5.2*1e-6, 5.2*10*1e-7 ?

print, 5.2e-6, 5.2*1e-6, 5.2*10*1e-7, format='(e20.10)'
5.2000000323e-06
5.1999995776e-06
5.2000000323e-06

Cheers,
Dominic

--
Dipl. Phys. Dominic R. Scales | Aero-Sensing Radarsysteme GmbH
Tel: +49 (0)8153-90 88 90 | c/o DLR Oberpfaffenhofen
Fax: +49 (0)8153-908 700 | 82234 Wessling, Germany
WWW: aerosensing.de | email: Dominic.Scales@aerosensing.de

Report message to a moderator

[Message index]

		rounding errors By: Dominic R. Scales on Fri, 27 April 2001 01:51
		Re: rounding errors By: Jaco van Gorkom on Fri, 04 May 2001 06:17
		Re: rounding errors By: Paul van Delst on Wed, 02 May 2001 09:46
		Re: rounding errors By: Dominic R. Scales on Wed, 02 May 2001 00:44
		Re: rounding By: thompson on Wed, 04 September 2002 07:59
		Re: rounding By: David Fanning on Wed, 04 September 2002 07:52
		Re: rounding By: Craig Markwardt on Wed, 04 September 2002 07:45
		Re: rounding By: Amar Nayegandhi on Wed, 04 September 2002 07:40

Previous Topic:	rounding
Next Topic:	Object Graphics Book.

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 02:22:47 PDT 2025

Total time taken to generate the page: 0.64248 seconds