| Re: Funny math? [message #20726] |
Mon, 24 July 2000 00:00 |
Bruce Bowler
Messages: 128
Registered: September 1998
|
Senior Member |
|
|
And of course there's the issue of not being able to represent certain
decimal numbers _exactly_ in the binary number system...
Simon de Vet wrote:
>
> I've got some arrays with data. None extend past 3 decimal places.
>
> 'print, filterfunc(0,0,half1), filterfunc(0,0,half2+200)'
> produces:
> '537.100 528.300'
>
> 'print, filterfunc(0,0,half1) - filterfunc(0,0,half2+200)'
> produces:
> '8.79999'
>
> What's up with that?
>
> Simon
|
|
|
|
| Re: Funny math? [message #20758 is a reply to message #20726] |
Thu, 20 July 2000 00:00  |
Alex Schuster
Messages: 124
Registered: February 1997
|
Senior Member |
|
|
Simon de Vet wrote:
> I've got some arrays with data. None extend past 3 decimal places.
>
> 'print, filterfunc(0,0,half1), filterfunc(0,0,half2+200)'
> produces:
> '537.100 528.300'
>
> 'print, filterfunc(0,0,half1) - filterfunc(0,0,half2+200)'
> produces:
> '8.79999'
>
> What's up with that?
It works better if you declare filterfunc as a DBLARR instead of a
FLTARR.
IDL> print, 537.1
537.100
This looks good, but the value is rounded. Internally, this float is
stored not exactly with 3 decimal places, so if you print more, you get
this output:
IDL> print, 537.1, format='(F12.8)'
537.09997559
With doubles instead of floats:
IDL> print, 537.1d
537.10000
More decimal places:
IDL> print, 537.1d format='(F12.8)'
537.10000000
Even more decimal places:
IDL> print, 537.1d, format='(F20.15)'
537.100000000000023
IDL> print, 537.1d - 528.3d
8.8000000
Alex
--
Alex Schuster Wonko@weird.cologne.de PGP Key available
alex@pet.mpin-koeln.mpg.de
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
