| Re: Time series with 75% missing observations [message #27344] |
Thu, 18 October 2001 09:00 |
Joe Means
Messages: 44
Registered: November 1996
|
Member |
|
|
Thanks much Craig. This should work great!
Cheers,
Joe
Craig Markwardt wrote:
> Joe Means <joe.means@orst.edu> writes:
>
>> I have long data series with many random, missing values. These series
>> each have only one or two frequencies. As I read the IDL 5.4
>> documentation, time series analysis routines require all values to be
>> evenly spaced. The periodicities are not sinusoidal, but such an
>> assumption might well find them. How can I find these periodicities?
>
>
> Greetings!
>
> If your time series are regularly sampled, but have missing values,
> then it is relatively straightforward to proceed with the FFT.
>
> Theoretically, zeroes do not actually contribute to the FFT power, but
> in practice you get aliases of the DC power which contaminate all the
> other frequencies. Contamination is bad.
>
> The solution is to subtract off the mean value of the signal *from the
> non-missing values* before doing the FFT. Say you have a variable Y
> which contains the signal, and missing values are set to -1. First
> you would transform to a new variable, YP, which has the average value
> subtracted.
>
> yp = y*0
> wh = where(y NE -1, ct)
> if ct EQ 0 then message, 'ERROR: no valid points!'
> yp(wh) = y(wh) - avg(y(wh))
>
> Note that the original missing values are converted to zeroes, so
> overall YP should have a zero mean value itself. Then you just do
> your FFT as normal.
>
> ***
>
> The other possibility, is if you have irregularly sampled points.
> Then you are better off with something like the Lomb Scargle
> periodogram. It's in Numerical Recipes. I have a crude routine which
> does this, available by request.
>
> Craig
>
>
|
|
|
|
| Re: Time series with 75% missing observations [message #27358 is a reply to message #27344] |
Thu, 18 October 2001 00:42  |
Roman Schreiber
Messages: 5
Registered: January 2001
|
Junior Member |
|
|
Craig Markwardt <craigmnet@cow.physics.wisc.edu> wrote in
news:onpu7lub39.fsf@cow.physics.wisc.edu:
>
> Joe Means <joe.means@orst.edu> writes:
>
>> I have long data series with many random, missing values. These series
>> each have only one or two frequencies. As I read the IDL 5.4
>> documentation, time series analysis routines require all values to be
>> evenly spaced. The periodicities are not sinusoidal, but such an
>> assumption might well find them. How can I find these periodicities?
>
> Greetings!
>
> If your time series are regularly sampled, but have missing values,
> then it is relatively straightforward to proceed with the FFT.
>
> Theoretically, zeroes do not actually contribute to the FFT power, but
> in practice you get aliases of the DC power which contaminate all the
> other frequencies. Contamination is bad.
>
> The solution is to subtract off the mean value of the signal *from the
> non-missing values* before doing the FFT. Say you have a variable Y
> which contains the signal, and missing values are set to -1. First
> you would transform to a new variable, YP, which has the average value
> subtracted.
>
> yp = y*0
> wh = where(y NE -1, ct)
> if ct EQ 0 then message, 'ERROR: no valid points!'
> yp(wh) = y(wh) - avg(y(wh))
>
> Note that the original missing values are converted to zeroes, so
> overall YP should have a zero mean value itself. Then you just do
> your FFT as normal.
>
> ***
>
> The other possibility, is if you have irregularly sampled points.
> Then you are better off with something like the Lomb Scargle
> periodogram. It's in Numerical Recipes. I have a crude routine which
> does this, available by request.
>
> Craig
>
>
Joe,
there is already Lomb periodogram function implemented in IDL 5.4
(LNP_TEST) based on routine fasper you may find in Numerical Recipes.
Best regards
Roman
--
************************************************************ **********
Roman Schreiber e-mail: schreibe@ncac.torun.pl
Copernicus Astronomical Centre PAS phone: 0-48-56-6219319
Astroph.Lab I fax: 0-48-56-6219381
ul.Rabianska 8
87-100 Torun POLAND
************************************************************ **********
|
|
|
|
| Re: Time series with 75% missing observations [message #27364 is a reply to message #27358] |
Wed, 17 October 2001 15:59 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Joe Means <joe.means@orst.edu> writes:
> I have long data series with many random, missing values. These series
> each have only one or two frequencies. As I read the IDL 5.4
> documentation, time series analysis routines require all values to be
> evenly spaced. The periodicities are not sinusoidal, but such an
> assumption might well find them. How can I find these periodicities?
Greetings!
If your time series are regularly sampled, but have missing values,
then it is relatively straightforward to proceed with the FFT.
Theoretically, zeroes do not actually contribute to the FFT power, but
in practice you get aliases of the DC power which contaminate all the
other frequencies. Contamination is bad.
The solution is to subtract off the mean value of the signal *from the
non-missing values* before doing the FFT. Say you have a variable Y
which contains the signal, and missing values are set to -1. First
you would transform to a new variable, YP, which has the average value
subtracted.
yp = y*0
wh = where(y NE -1, ct)
if ct EQ 0 then message, 'ERROR: no valid points!'
yp(wh) = y(wh) - avg(y(wh))
Note that the original missing values are converted to zeroes, so
overall YP should have a zero mean value itself. Then you just do
your FFT as normal.
***
The other possibility, is if you have irregularly sampled points.
Then you are better off with something like the Lomb Scargle
periodogram. It's in Numerical Recipes. I have a crude routine which
does this, available by request.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
