comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: Funny math?

Re: Funny math?
type: item
description: Simon de Vet wrote: > I've got some arrays with data. None extend past 3 decimal places. > > 'print, filterfunc(0,0,half1), filterfunc(0,0,half2+200)' > produces: > '537.100 528.300' > > 'print, filterfunc(0,0,half1) - filterfunc(0,0,half2+200)' > produces: > '8.79999' > > What's up with that? It works better if you declare filterfunc as a DBLARR instead of a FLTARR. IDL> print, 537.1 537.100 This looks good, but the value is rounded. Internally, this float is stored not exactly with 3 decimal places, so if you print more, you get this output: IDL> print, 537.1, format='(F12.8)' 537.09997559 With doubles instead of floats: IDL> print, 537.1d 537.10000 More decimal places: IDL> print, 537.1d format='(F12.8)' 537.10000000 Even more decimal places: IDL> print, 537.1d, format='(F20.15)' 537.100000000000023 IDL> print, 537.1d - 528.3d 8.8000000 Alex -- Alex Schuster Wonko@weird.cologne.de PGP Key available alex@pet.mpin-koeln.mpg.de
creator: Alex Schuster
date: 2000-07-20T07:00:00-00:00

Re: Funny math?
type: item
description: And of course there's the issue of not being able to represent certain decimal numbers _exactly_ in the binary number system... Simon de Vet wrote: > > I've got some arrays with data. None extend past 3 decimal places. > > 'print, filterfunc(0,0,half1), filterfunc(0,0,half2+200)' > produces: > '537.100 528.300' > > 'print, filterfunc(0,0,half1) - filterfunc(0,0,half2+200)' > produces: > '8.79999' > > What's up with that? > > Simon
creator: Bruce Bowler
date: 2000-07-24T07:00:00-00:00