| Filtering out noise quickly? [message #91346] |
Wed, 01 July 2015 15:41  |
wdolan
Messages: 29
Registered: June 2015
|
Junior Member |
|
|
So I've got a set of frequency data and a set of 'residual' data, and I am plotting the frequency on the X and the residual on the Y. We are looking to see if a trough in the residuals at a certain frequency is present. But there is a whole bunch of noise in the residuals, so I am trying to write a program which will detect if the trough is there or if it is just noise. As of now I have something that works ok. It calculates the average and standard deviation for the data set as a whole (not including the frequency we want), and then calculates the average for the small range of the frequencies where we are looking for the trough. Next it "buffers" (that probably isn't the right terminology..), the average for the whole thing with the standard deviation. So if the average value at the good frequency is less than the average general minus the general standard deviation, we should have a trough and not noise. Hopefully that makes sense, please ask if you have questions.
Is there any sort of tool set of procedures that could eliminate the noise in an easier way?
|
|
|
|
| Re: Filtering out noise quickly? [message #91350 is a reply to message #91346] |
Thu, 02 July 2015 00:59  |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
On Wednesday, July 1, 2015 at 3:41:28 PM UTC-7, wdo...@oxy.edu wrote:
> So I've got a set of frequency data and a set of 'residual' data, and I am plotting the frequency on the X and the residual on the Y. We are looking to see if a trough in the residuals at a certain frequency is present. But there is a whole bunch of noise in the residuals, so I am trying to write a program which will detect if the trough is there or if it is just noise. As of now I have something that works ok. It calculates the average and standard deviation for the data set as a whole (not including the frequency we want), and then calculates the average for the small range of the frequencies where we are looking for the trough. Next it "buffers" (that probably isn't the right terminology..), the average for the whole thing with the standard deviation. So if the average value at the good frequency is less than the average general minus the general standard deviation, we should have a trough and not noise. Hopefully that makes sense, please ask if you have questions.
>
> Is there any sort of tool set of procedures that could eliminate the noise in an easier way?
What you want to do is a statistical hypothesis test. This could be a chi-square test. You would start with an expected signal, i.e. a model of the signal behavior. The null hypothesis is that the expected signal is not there, i.e. the data set is consistent with random noise.
To say more, one would need to know more about the expected signal, whether it has a width in frequency or not, whether it has a particular shape, and whether it always comes at a known frequency or the center frequency could vary. Those criteria, among others would help determine how sophisticated a process is required.
CM
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
