| EOF analysis of large data with NAN's [message #88313] |
Sat, 12 April 2014 16:19  |
anil
Messages: 34
Registered: August 2009
|
Member |
|
|
Hi,
I am trying to conduct Empirical Orthogonal Function analysis on satellite data, which is highly covered by clouds. I tried to follow:
https://www.idlcoyote.com/code_tips/eof_analysis.html
but it seems I am doing something wrong.
I have temperature data (Avhrr- Satellite sea surface temperature) from 1981-2009, which is monthly. My space grid is 346X138.
So I do have 340 months, 346 longitudes and 138 latitudes, which makes a large array of 340x47748. Plus the data is highly invaded with not a number values.
Therefore I can not calculate the Covariance matrix using the ## operator. I get an error saying:
Unable to allocate memory: to make array.
Cannot allocate memory
So what I try to make a 47748x47748 operation which does not work (this is normal I guess :) ).
When I use the # operator only, it calculates a covariance matrix (I think it does a 340x340 this time). Then when I try to do:
LA_SVD, matrix, W, U, V
Then I get :
Loaded DLM: LAPACK.
% LA_SVD: Routine did not converge, STATUS=20.
I read about EOF analysis, but I could not figure out how to carry it out on a 340(time)x47748(space) dataset. Could it be NaN's that cause the problem? They are assigned to a certain value and set as Nan's in my code.
Could anyone please tell me what I am doing wrong?
Best,
Anil
|
|
|
|
|
|
| Re: EOF analysis of large data with NAN's [message #89853 is a reply to message #89851] |
Thu, 11 December 2014 01:30  |
anil
Messages: 34
Registered: August 2009
|
Member |
|
|
Thank you for your answer. But unfortunately the result is pretty much the same SVDC: svdcmp_f did not converge. I think it is not possible to compute without filling the NaN's (which are quite a lot!) or somehow removing them. Removing does not seem to be the correct way (as far as I understand from the documents I read on EOF/PCA with missing values). I will proceed with a filling algorithm. I think i will try DINEOF which seems to be a proven way of carrying out EOF with lots of NaN's.
thanks anyway.
Anil
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
