| Re: "Correct" Data Philosophy [message #69179 is a reply to message #69024] |
Sat, 19 December 2009 06:52   |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"Kenneth P. Bowman" <k-bowman@null.edu> wrote in message
news:k-bowman-565924.17033817122009@news.tamu.edu...
> In article <MPG.2594571640f8a8219896ab@news.giganews.com>,
> David Fanning <news@dfanning.com> wrote:
>
>> OK, here is my problem: I don't have any idea what you
>> people are talking about. And neither do the folks asking
>> me questions. :-(
>
> That crux of the issue here is that this problem is *hard*,
> and it is difficult to generalize from one situation to
> another. Kind of like asking -- "How do I write a good IDL
> program?" :-)
>
> Experience with similar data sets is very helpful -- that is,
> we learn by doing (and making mistakes and re-doing).
>
>>
>> This, in particular, is opaque to me:
>>
>> If you need to do a Fourier transform, consider using
>> least-squares estimation rather than interpolating
>> and using an FFT.
>>
>> OK, I will, but *how*!?
>
> This is actually quite easy. You can use REGRESS. I'll try to
> write a short example that will demonstrate, among other things,
> that when there is *no* missing data, least squares is exactly
> equivalent to the FFT.
This is true. However, the moment you remove even one point,
all the columns off the matrix ( i.e. Ax=b, where b is the data,
and x is the spectrum) are no longer orthogonal, and thus
one cannot use x = A^t b (which is the fft).
We are now stuck with using x = (A^tA)^-1 A^t b
which requires many many more calculations, and in my
experience, is most often an illposed matrix.
Even greatly reducing the number of spectral points, in order
to ensure an overdetermined system, did not lead to reliable results.
gaps are problematic to spectral analysis. It is basically a divide
by zero problem. Your starting point is the true spectrum convolved
with a spectrum of your gap funtion (i.e. 1's and 0's), and there is no
good way to deconvolve that.
A cimmon solution I have used has been to interpolate the gaps, perform a
local
spectral analysis (using the S-Transform for instance :), and
then reinserting these gaps into that local spectrum.
cheers,
bob
PS lomb scargle is not an adequate solution to the problem.
It calculates _one_ spectral component in a least squares manner,
and should not be used to calculate the full spectrum.
Numerical Recipes screwed that one up.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
