comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: Re: Roundoff error in SMOOTH

Re: Roundoff error in SMOOTH
type: item
description: landsman@stars.gsfc.nasa.gov (Wayne Landsman) writes: > I was recently surprised to discover that applying the SMOOTH function to a > non-negative array could yield an array with negative numbers. I give an > example below. This problem is evidently due to some sort of roundoff error, > since it does not occur when using double precision. But it is not obvious to > me how averaging 9 non-negative numbers (for a 3x3 box smooth) could yield a > negative number, even allowing for roundoff error. > Although not obvious from my simple example, this has nothing to do with edge > effects -- I originally found the problem when 3x3 smoothing a 1024 x 1024 > array. (rest deleted) My theory is that the SMOOTH function probably uses Fourier transforms to speed up the convolution, and this is where the round-off error is sneaking in. Bill Thompson
creator: thompson
date: 1999-05-26T07:00:00-00:00

Re: Roundoff error in SMOOTH
type: item
description: In article <26MAY199915495025@stars.gsfc.nasa.gov>, landsman@stars.gsfc.nasa.gov (Wayne Landsman) writes... > I was recently surprised to discover that applying the SMOOTH function to a > non-negative array could yield an array with negative numbers. I give an > example below. I forgot to include the IDL !VERSION used in my example: { sparc sunos unix 5.2 Oct 30 1998} Testing the same example on VMS IDL V4.0.1, I still get negative numbers although not as many! --Wayne Landsman landsman@mpb.gsfc.nasa.gov
creator: landsman
date: 1999-05-26T07:00:00-00:00