| Empirical Orthogonal Function Analysis in IDL [message #60023] |
Sat, 26 April 2008 20:01  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Folks,
I've been spending my time the past several weeks learning
the ins and outs of Empirical Orthogonal Function (EOF)
analysis. This is a multivariate statistical technique,
identical to Principal Component analysis, which was
developed in the 1950s by Edward Lorenz, of the famous
"butterfly effect". Dr. Lorenz, a Boulder resident, just
died a week or two ago.
In any case, this is something we do quite a lot around
the shop where I am now working, and I inherited some
code I didn't really understand, so I started to write
my own code, mostly as a way to understand the technique.
The old code typically took hours, and in some cases, days
to run.
But in the course of writing my own, I stumbled onto a
mathematical trick that allowed me to produce identical
results compared to the old way in about four tenths of a
second! Wow! Big breakthrough.
I don't take credit for the trick (I found it in Wilks
outstanding book, Statistical Methods in the Atmospheric
Sciences) and it took about three of us, working together,
to produce the serendipity needed to come to the realization
of what we were doing. But it is definitely worth knowing
about.
So I've written an article that outlines the essential
steps of the process. It is available here:
http://www.dfanning.com/code_tips/eof_analysis.html
Please let me know if you have any insights to add to this
process. I can't say I know everything there is to know
about this subject, but I am extremely happy with the
code I have to do this now.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| Re: Empirical Orthogonal Function Analysis in IDL [message #60169 is a reply to message #60023] |
Sun, 27 April 2008 10:10  |
d.poreh
Messages: 406
Registered: October 2007
|
Senior Member |
|
|
David
It is very interesting to me. I’ve never heard about this method. I
have another problem now:
How we can extract the hottest point in the world based on this data
(for each year we have a maximum temperature somewhere) and plot this
on the world map (with a projection) for 1969 until now?
Cheers
On Apr 27, 5:01 am, David Fanning <n...@dfanning.com> wrote:
> Folks,
>
> I've been spending my time the past several weeks learning
> the ins and outs of Empirical Orthogonal Function (EOF)
> analysis. This is a multivariate statistical technique,
> identical to Principal Component analysis, which was
> developed in the 1950s by Edward Lorenz, of the famous
> "butterfly effect". Dr. Lorenz, a Boulder resident, just
> died a week or two ago.
>
> In any case, this is something we do quite a lot around
> the shop where I am now working, and I inherited some
> code I didn't really understand, so I started to write
> my own code, mostly as a way to understand the technique.
> The old code typically took hours, and in some cases, days
> to run.
>
> But in the course of writing my own, I stumbled onto a
> mathematical trick that allowed me to produce identical
> results compared to the old way in about four tenths of a
> second! Wow! Big breakthrough.
>
> I don't take credit for the trick (I found it in Wilks
> outstanding book, Statistical Methods in the Atmospheric
> Sciences) and it took about three of us, working together,
> to produce the serendipity needed to come to the realization
> of what we were doing. But it is definitely worth knowing
> about.
>
> So I've written an article that outlines the essential
> steps of the process. It is available here:
>
> http://www.dfanning.com/code_tips/eof_analysis.html
>
> Please let me know if you have any insights to add to this
> process. I can't say I know everything there is to know
> about this subject, but I am extremely happy with the
> code I have to do this now.
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| RE: Empirical Orthogonal Function Analysis in IDL [message #87428 is a reply to message #87417] |
Sat, 01 February 2014 05:08 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
collinan writes:
> When running your EOF code, I had a problem due to a mask overlaid on the time dataset.
> I've firstly reformed the matrix devoid of the NaN values.
> However, I could not then reform (the inverse reform) the first EOF since it was not the same dimensions as the initial masked dataset.
>
> Do you have any tips for that?
It has been a very long time since I used these EOF programs, but I have
to admit, I have no idea how the situation you describe could possibly
occur. I think there must be a high probability you have simply made a
mistake in your coding. Sorry I don't have better advice.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
