comp.lang.idl-pvwave archive: archive » How can one improve the derivative?

Home » Public Forums » archive » How can one improve the derivative?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Switch to threaded view of this topic

Create a new topic

Submit Reply

How can one improve the derivative? [message #86662]

Wed, 27 November 2013 04:25

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

Hi

I had to differentiate noisy data - which is a problem. I was wondering, how can one improve the derivative?
I read on a blog that one way is to make a Fourier filtering afterwards and another way is to apply a "smoothed" or "filtered" gradient. But I had no idea how to do this in IDL. Can anyone help?

Thanks in advance

Report message to a moderator

Re: How can one improve the derivative? [message #86664 is a reply to message #86662]

Wed, 27 November 2013 05:26

David Fanning is currently offline

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

g.nacarts@gmail.com writes:

> I had to differentiate noisy data - which is a problem. I was wondering, how can one improve the derivative?
> I read on a blog that one way is to make a Fourier filtering afterwards and another way is to apply a "smoothed" or "filtered" gradient. But I had no idea how to do this in IDL. Can anyone help?

I think I would just apply the SMOOTH function to your data before you
differentiate it.

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86667 is a reply to message #86664]

Wed, 27 November 2013 06:46

wlandsman is currently offline

wlandsman
Messages: 743
Registered: June 2000

Senior Member

You might look at the SAVGOL function which gets rid of the high-frequency signal (noise) while preserving the lower-frequency signals. (Smooth() suppresses both high and low frequencies.) The documentation for SAVGOL() says

"Tip: You can use this function in conjunction with the CONVOL function for smoothing and optionally for numeric differentiation."

On Wednesday, November 27, 2013 8:26:08 AM UTC-5, David Fanning wrote:
> g.nacarts@gmail.com writes:
>
>
>
>> I had to differentiate noisy data - which is a problem. I was wondering, how can one improve the derivative?
>
>> I read on a blog that one way is to make a Fourier filtering afterwards and another way is to apply a "smoothed" or "filtered" gradient. But I had no idea how to do this in IDL. Can anyone help?
>
>
>
> I think I would just apply the SMOOTH function to your data before you
>
> differentiate it.
>
>
>
> Cheers,
>
>
>
> David
>
> --
>
> David Fanning, Ph.D.
>
> Fanning Software Consulting, Inc.
>
> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
>
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86668 is a reply to message #86662]

Wed, 27 November 2013 06:50

lecacheux.alain is currently offline

lecacheux.alain
Messages: 325
Registered: January 2008

Senior Member

Le mercredi 27 novembre 2013 13:25:06 UTC+1, g.na...@gmail.com a écrit :
> Hi
>
>
>
> I had to differentiate noisy data - which is a problem. I was wondering, how can one improve the derivative?
>
> I read on a blog that one way is to make a Fourier filtering afterwards and another way is to apply a "smoothed" or "filtered" gradient. But I had no idea how to do this in IDL. Can anyone help?
>
>
>
> Thanks in advance

Differentiating noisy data can often be obtained by convolution with a Savitzky-Golay filter. In IDL, look at SAVGOL and CONVOL functions.

alx.

Report message to a moderator

Re: How can one improve the derivative? [message #86680 is a reply to message #86668]

Thu, 28 November 2013 04:28

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

Hi

I read the SAVGOL function but there is something I didn't understand very well. How can I choose the nL and nR?

Report message to a moderator

Re: How can one improve the derivative? [message #86681 is a reply to message #86680]

Thu, 28 November 2013 05:06

lecacheux.alain is currently offline

lecacheux.alain
Messages: 325
Registered: January 2008

Senior Member

Le jeudi 28 novembre 2013 13:28:40 UTC+1, g.na...@gmail.com a écrit :
> Hi
>
>
>
> I read the SAVGOL function but there is something I didn't understand very well. How can I choose the nL and nR?

Afaik, there is no definite rule. At order 0, the Savitzky-Golay filter is a smoothing filter of length nL+nR. The rule of thumb would be to keep the filter width not much larger than the narrowest feature you want to preserve. At higher order, it is a derivative filter. The filtering rule should be the same.
The best way is to try...
alx.

Report message to a moderator

Re: How can one improve the derivative? [message #86684 is a reply to message #86681]

Thu, 28 November 2013 05:27

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

This is what I did, I was playing with different values for nL and nR. But I was wondering if there is any specific rule. Also, when I used order=1 to give me the first order derivative filter the answer was not converges at all.
At order 0, the Savitzky-Golay filter seems to make the things better but not much.

Report message to a moderator

Re: How can one improve the derivative? [message #86685 is a reply to message #86684]

Thu, 28 November 2013 05:35

David Fanning is currently offline

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

g.nacarts@gmail.com writes:

> This is what I did, I was playing with different values for nL and nR. But I was wondering if there is any specific rule. Also, when I used order=1 to give me the first order derivative filter the answer was not converges at all.
> At order 0, the Savitzky-Golay filter seems to make the things better but not much.

There is a good example of high frequency filtering with the FFT
function in the Image Processing chapter of Coyote's Guide to
Traditional IDL Graphics.

I found three of these books in the back of the closet, in case you are
interested. These are the last three. I'm providing free shipping in the
US while they last.

http://www.idlcoyote.com/coyotestore/

Cheers,

David

--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86686 is a reply to message #86685]

Thu, 28 November 2013 06:22

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

I have already this book. I am doing simulations so how can I use the example of FFT function?

Report message to a moderator

Re: How can one improve the derivative? [message #86687 is a reply to message #86686]

Thu, 28 November 2013 06:27

David Fanning is currently offline

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

g.nacarts@gmail.com writes:

> I have already this book. I am doing simulations so how can I use the example of FFT function?

Well, write code that applies a low-pass filter. It's on page 285 in my
book. ;-)

Cheers,

David

--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86688 is a reply to message #86687]

Thu, 28 November 2013 06:42

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

> Well, write code that applies a low-pass filter. It's on page 285 in my book. ;-)

You mean the smooth function right? I used the smooth function with the keyword /Edge_Truncate as shown in your example but the results I got are not satisfactory. I was trying to perform image filtering with the Convol function but I didn't understand how to choose the kernel.

Report message to a moderator

Re: How can one improve the derivative? [message #86689 is a reply to message #86688]

Thu, 28 November 2013 06:48

David Fanning is currently offline

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

g.nacarts@gmail.com writes:

> You mean the smooth function right? I used the smooth function with the keyword /Edge_Truncate as shown in your example but the results I got are not satisfactory. I was trying to perform image filtering with the Convol function but I didn't understand how to choose the kernel.

No, I meant try a low-pass filter using frequency filtering with the FFT
function to filter out the high-frequency noise components.

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86694 is a reply to message #86689]

Thu, 28 November 2013 07:08

g.nacarts is currently offline

g.nacarts
Messages: 148
Registered: November 2013

Senior Member

> No, I meant try a low-pass filter using frequency filtering with the FFT function to filter out the high-frequency noise components.

I am completely confused. Can you please give an example what you mean?
The syntax of FFT : smoothresult=FFT(Array) is that what you mean?

Report message to a moderator

Re: How can one improve the derivative? [message #86695 is a reply to message #86694]

Thu, 28 November 2013 07:12

David Fanning is currently offline

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

g.nacarts@gmail.com writes:

> I am completely confused. Can you please give an example what you mean?
> The syntax of FFT : smoothresult=FFT(Array) is that what you mean?

I've given an example. You have the book, right? I'm looking at page 296
in the section labeled Image Filtering in the Frequency Domain.

Cheers,

David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: How can one improve the derivative? [message #86701 is a reply to message #86684]

Thu, 28 November 2013 08:53

lecacheux.alain is currently offline

lecacheux.alain
Messages: 325
Registered: January 2008

Senior Member

Le jeudi 28 novembre 2013 14:27:41 UTC+1, g.na...@gmail.com a écrit :
> This is what I did, I was playing with different values for nL and nR. But I was wondering if there is any specific rule. Also, when I used order=1 to give me the first order derivative filter the answer was not converges at all.
>
> At order 0, the Savitzky-Golay filter seems to make the things better but not much.

SG filtering assumes that your function can be locally fitted by a low order polynomial (of order 2 to 4, at most nL+nR). The (LS) fit is done for each point of the function. The output of the filter is the constant term of the best polynomial fit. This is similar to a moving average, except that the latter uses a constant (a polynomial of degree 0). Derivative is obtained in the same way, but the constant term of the corresponding derived polynomial is used in that case.
Everything will depend on the shape of your function. Derivatives noisier than
the function is a normal result: differenciating increases noise while integrating is a smoothing process.
alx.

Report message to a moderator

Switch to threaded view of this topic

Create a new topic

Submit Reply

Previous Topic:	ENVI_OPEN_FILE not found
Next Topic:	black wrong multicolor map

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

PDF

]

Current Time: Wed Oct 08 13:34:40 PDT 2025

Total time taken to generate the page: 0.00550 seconds