Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
Hi, I am trying to conduct Empirical Orthogonal Function analysis on satellite data, which is highly covered by clouds. I tried to follow: https://www.idlcoyote.com/code_tips/eof_analysis.html but it seems I am doing something wrong. I have temperature data (Avhrr- Satellite sea surface temperature) from 1981-2009, which is monthly. My space grid is 346X138. So I do have 340 months, 346 longitudes and 138 latitudes, which makes a large array of 340x47748. Plus the data is highly invaded with not a number values. Therefore I can not calculate the Covariance matrix using the ## operator. I get an error saying: Unable to allocate memory: to make array. Cannot allocate memory So what I try to make a 47748x47748 operation which does not work (this is normal I guess :) ). When I use the # operator only, it calculates a covariance matrix (I think it does a 340x340 this time). Then when I try to do: LA_SVD, matrix, W, U, V Then I get : Loaded DLM: LAPACK. % LA_SVD: Routine did not converge, STATUS=20. I read about EOF analysis, but I could not figure out how to carry it out on a 340(time)x47748(space) dataset. Could it be NaN's that cause the problem? They are assigned to a certain value and set as Nan's in my code. Could anyone please tell me what I am doing wrong? Best, Anil
Re: EOF analysis of large data with NAN's
You should use the Singular Value Decomposition (SVD) method instead of the covariance/correlation matrix method. The latter is computationally impractical for large, regularly spaced data fields such as a sequence of satellite images. Details over SVD method can be found on Section 4.2 of the Manual for EOF and SVD Analyses of Climate Data by Björnsson and Venegas. michellemota -- http://compgroups.net/comp.lang.idl-pvwave/
Thank you for your answer. But unfortunately the result is pretty much the same SVDC: svdcmp_f did not converge. I think it is not possible to compute without filling the NaN's (which are quite a lot!) or somehow removing them. Removing does not seem to be the correct way (as far as I understand from the documents I read on EOF/PCA with missing values). I will proceed with a filling algorithm. I think i will try DINEOF which seems to be a proven way of carrying out EOF with lots of NaN's. thanks anyway. Anil