| Re: Seasonal Variation in Trend Analysis [message #56034] |
Mon, 01 October 2007 12:16 |
wita
Messages: 43
Registered: January 2005
|
Member |
|
|
> When I initially ran the example program in cgi_fftndvi.pro
> I got thrown into an infinite loop. I killed the process and
> moved the CATCH, /CANCEL in your error handler to the *first*
> line after the CATCH, and then the program ran perfectly.
Thanks for the tip, I wil change this in the next version of HANTS.
best regards,
Allard
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56048 is a reply to message #56034] |
Fri, 28 September 2007 09:35  |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"David Fanning" <news@dfanning.com> wrote in message
news:MPG.2166d611a2078c5498a0a0@news.frii.com...
> Allard de Wit writes:
>
>> Anyway try this one: ftp://sc:image1@ftp.alterra.nl/pub/adewit/hants.zip
>
> Wow! This has even caused me to go dig out that E. Oran Brigham book
> on the FFT that Bob suckered me into buying several years ago.
:O
"suckered"? That is one of the best books of all time!
Better than War and Peace, and Lord of the Rings combined.
Rumour has it that Angelina Jolie is in talks to play DFT in
the movie version of it.
Seriously, anyone who has ever used the fft() routine must
read that book first (imho) :)
Cheers,
bob
|
|
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56052 is a reply to message #56049] |
Fri, 28 September 2007 07:34 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
wita writes:
> You may want to have a look at the IDL implementation of the HANTS
> (Harmonic Analyses of NDVI Time Series) algorithm I wrote. The code is
> available at: http://www.ittvis.com/codebank/search.asp?FID=333
>
> Basically HANTS searches for the seasonality rather then removing it,
> but the principle is the same.
Thank you. This is very helpful.
When I initially ran the example program in cgi_fftndvi.pro
I got thrown into an infinite loop. I killed the process and
moved the CATCH, /CANCEL in your error handler to the *first*
line after the CATCH, and then the program ran perfectly.
To test, I set it back where it was before, and the program
*still* ran fine! So, I don't know what that is all about.
But I do feel more comfortable having that CANCEL as the
first line after a CATCH to avoid these kinds of problems. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56069 is a reply to message #56068] |
Thu, 27 September 2007 12:09 |
wita
Messages: 43
Registered: January 2005
|
Member |
|
|
David,
>
> Well, I can see this is what I *want* to do, but I
> am still having some trouble with the *how* to do. :-)
You may want to have a look at the IDL implementation of the HANTS
(Harmonic Analyses of NDVI Time Series) algorithm I wrote. The code is
available at: http://www.ittvis.com/codebank/search.asp?FID=333
Basically HANTS searches for the seasonality rather then removing it,
but the principle is the same.
with best regards,
Allard
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56070 is a reply to message #56069] |
Thu, 27 September 2007 10:26 |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"David Fanning" <news@dfanning.com> wrote in message
news:MPG.21655a1d1487d44b98a09d@news.frii.com...
> Ingo von Borstel writes:
>
>> I'd do a Fourier transform, filter out the terms that correspond to the
>> seasonal frequency/ies and possibly do a back-transform.
>
> Well, I can see this is what I *want* to do, but I
> am still having some trouble with the *how* to do. :-)
>
> For example, how, exactly, would I construct such
> a filter. To knock out the first harmonic, would
> I construct a circular filter near the origin? How
> near?
As a quick answer, FFT the data (assuming regular sampling etc)
and find the frequency corresponding to the seasonal signal.
(frequencies are i/NT where T is the sampling interval - for instance
one measurement each day).
Multiply your spectrum with zero at that point (and 1 every other point)
(also do that for the negative frequency component - assuming real data).
Invert the FFT.
You have removed the precise seasonal frequency.
You can perhaps improve that by multiplying the adjacent frequencies
by 0.5 above (i.e. your mask is 1,1,1,1,1,1....1,1 ,0.5,0,0.5,1,1.....)
That is equivalent to applying a hanning window filter to remove the
seasonal signal.
Sorry for brief description, I am taking care of a 2.5 yr old with the flu
right now so I have to be quick.
Cheers,
bob
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56072 is a reply to message #56070] |
Thu, 27 September 2007 06:13 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Kenneth P. Bowman writes:
> There are several ways to do this, David, depending on exactly what
> you are trying to do.
Thanks, Ken. Very helpful. I appreciate it. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56073 is a reply to message #56072] |
Thu, 27 September 2007 06:12 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Ingo von Borstel writes:
> I'd do a Fourier transform, filter out the terms that correspond to the
> seasonal frequency/ies and possibly do a back-transform.
Well, I can see this is what I *want* to do, but I
am still having some trouble with the *how* to do. :-)
For example, how, exactly, would I construct such
a filter. To knock out the first harmonic, would
I construct a circular filter near the origin? How
near? I take it from both your post and Bob's that
this is pretty much common knowledge, but if it is,
I must have been asleep in that class. The books I
have been using to understand regression and trend
analysis have not mentioned FFT filtering at all.
And although I think I have at least a limited grasp
of the theory, a practical understanding of the IDL
code to implement it still eludes me.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56074 is a reply to message #56073] |
Wed, 26 September 2007 20:01 |
Kenneth P. Bowman
Messages: 585
Registered: May 2000
|
Senior Member |
|
|
In article <MPG.2164877cbc85365998a09c@news.frii.com>,
David Fanning <news@dfanning.com> wrote:
> Folks,
>
> Does anyone happen to have an IDL example of some
> code that might remove seasonal variation in a time
> series? Or some suggestions for how to proceed in IDL?
> I can see that I might want to use a model that has
> sin and cosine terms, but I can't see how to find the
> coefficients of such a model in IDL.
There are several ways to do this, David, depending on exactly what
you are trying to do.
Let's assume that you have evenly-spaced data, say monthly,
and you have complete years of data (12*nyears data points).
Those assumptions are not essential, but make the analysis
simpler.
One approach is to remove the climatological annual cycle. This is
done by computing the average of all the Januaries, all the Februaries,
etc. Then subtract the average January from each of the individual
Januaries. The residual is called the "anomaly".
Another approach is to remove the annual harmonic (a perfect
sinusoid). This can be done with an FFT, but is usually easier to do
with simple regression. (FFT and regression are equivalent for
evenly-spaced data with no missing values.)
The difference between these two methods is that the second
method removes only the first harmonic of the annual cycle.
The first method removes the first *six* harmonics. In many
cases the annual cycle is dominated by the first harmonic,
so the results of the two methods are similar. In some cases, the
semi-annual harmonic might be important, in which case the second method
would not work as well. The second method, however, guarantees
a smooth estimate of the annual cycle. (Note that you could also compute
the higher harmonics, with FFT or REGRESS, and then remove as many
harmonics as you want.)
Here is a sample program that illustrates both approaches using some
synthetic monthly data. The values are assumed to be located at the
centers of the months.
Cheers, Ken
; Removing the climatological mean
nyears = 4 ;Number of years
nmonths = 12*nyears ;Number of months
t = (0.5 + FINDGEN(nmonths))/12 ;Time in years
z = COS(2.0*!PI*t) + 0.2*RANDOMN(seed, n) ;Data with an annual cycle
z = REFORM(z, 12, nyears) ;Rearrange for convenience
zclim = REBIN(TOTAL(z, 2)/nyears, 12, nyears) ;Compute climatological mean
PLOT, t, z, PSYM = 1 ;Plot original data
OPLOT, t, zclim ;Plot climatological mean
OPLOT, t, z - zclim, LINESTYLE = 1 ;Plot anomaly
cr = ''
READ, cr, PROMPT = 'Enter <cr> to continue : '
; Removing the annual harmonic
z = COS(2.0*!PI*t - 0.75) + 0.2*RANDOMN(seed, n) ;Data with an annual cycle
x = TRANSPOSE([[SIN(2.0*!PI*t)], [COS(2.0*!PI*t)]]) ;Create predictor variables
coeffs = REGRESS(x, z, CONST = zmean, YFIT = yfit) ;Fit the sinusoids
PRINT, SQRT(TOTAL(coeffs^2)), ATAN(coeffs[0], coeffs[1]) ;Amplitude and phase
PLOT, t, z, PSYM = 1 ;Plot original data
OPLOT, t, yfit ;Plot climatological mean
OPLOT, t, z - yfit, LINESTYLE = 1 ;Plot anomaly
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56075 is a reply to message #56074] |
Thu, 27 September 2007 05:04 |
Ingo von Borstel
Messages: 54
Registered: September 2006
|
Member |
|
|
Hi,
>
> Does anyone happen to have an IDL example of some
> code that might remove seasonal variation in a time
> series? Or some suggestions for how to proceed in IDL?
> I can see that I might want to use a model that has
> sin and cosine terms, but I can't see how to find the
> coefficients of such a model in IDL.
I'd do a Fourier transform, filter out the terms that correspond to the
seasonal frequency/ies and possibly do a back-transform.
Best regards,
Ingo
--
Ingo von Borstel <newsgroups@planetmaker.de>
Public Key: http://www.planetmaker.de/ingo.asc
If you need an urgent reply, replace newsgroups by vgap.
|
|
|
|
| Re: Seasonal Variation in Trend Analysis [message #56077 is a reply to message #56074] |
Wed, 26 September 2007 15:42 |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"David Fanning" <news@dfanning.com> wrote in message
news:MPG.2164877cbc85365998a09c@news.frii.com...
> Folks,
>
> Does anyone happen to have an IDL example of some
> code that might remove seasonal variation in a time
> series? Or some suggestions for how to proceed in IDL?
> I can see that I might want to use a model that has
> sin and cosine terms, but I can't see how to find the
> coefficients of such a model in IDL.
>
> Cheers,
>
> David
Just Least Squares fit the cos and sine functions (0 phase in both
those, you get phase from the amplitudes).
Ax=b
where
[c,s] [x] = [data]
x is the unknown. c, s are your sine and cosine terms.
Use the choledky (faster) or SVD solver routines.
interesting note, if the total length of the time series is a
multiple of the length of the season, just FFT and read off those
values (real = cos, imag = sin for the appropriate frequency)
It is a 'fast' way of solving the least squares fit problem
Cheers,
bob
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
