Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
Hi everyone! help me, please! my IDL code has 3 nested for (for a 256x256x60matrix).estimated time for the result is 21hours approximately, but after 12 hours my work crashed! if I decrease matrix dim., the result is ok! do you think the problem is my pc? do you have an alternative idea for my "infinite for"? Cheers, M.C.
Am 25.11.2013 09:13, schrieb Chrisss: > my IDL code has 3 nested for (for a 256x256x60matrix).estimated time > for the result is 21hours approximately, but after 12 hours my work > crashed! if I decrease matrix dim., the result is ok! do you think > the problem is my pc? do you have an alternative idea for my > "infinite for"? To say more about that, some code is needed. What calculations are within the loops? Please provide a code example! Anyway, a short vision from my crystal ball told me that for-loops may be evil incarnate -> http://www.idlcoyote.com/tips/forloops.html ;-) Cheers, Kai
evil? oh no!!! :-( this is for-loop: for i=0, dx-1 do begin for j=0,dy-1 do begin y=reform (Im[i,j,*],N_ELEMENTS(x))) fitErrors=FLTARR(N_ELEMENTS(x)) T1values=FLTARR(N_ELEMENTS(x)) S0values=FLTARR(N_ELEMENTS(x)) fvalues=FLTARR(N_ELEMENTS(x)) for t=0,N_ELEMENTS(x)-1 do begin T0=t yCurr=y if (t ne 0) then yCurr[0:T0]=-y[0:To] A=mpfitfun('myfunct', x,yCurr, ERR, PARINFO=parinfo, WEIGHTS=w, YFIT=yfit, $ BESTNORM=error, /QUIET ) fitErrors[t]=error T1values[t]=A[0] S0values[t]=A[1] fvalues[t]=A[2] endfor min=MIN(fitErrors,idmin) T1=T1values[idmin] f=fvalues[idmin] S0=S0values[idmin] yfit=myfunct(x,[S0,f,T1]) plot,x,y oplot,x,yfit oplot,x,abs(yfit) wait,1 T1map[i,j]=T1 endfor endfor where x=t;t=t*(findgen(60)+1) dx=dy=256, Im is a 256x256x60 matrix cheers, MC
Am 25.11.2013 11:24, schrieb Chrisss: > evil? oh no!!! :-( > this is for-loop: > > for i=0, dx-1 do begin > for j=0,dy-1 do begin > y=reform (Im[i,j,*],N_ELEMENTS(x))) > fitErrors=FLTARR(N_ELEMENTS(x)) > T1values=FLTARR(N_ELEMENTS(x)) > S0values=FLTARR(N_ELEMENTS(x)) > fvalues=FLTARR(N_ELEMENTS(x)) > for t=0,N_ELEMENTS(x)-1 do begin > T0=t > yCurr=y > if (t ne 0) then yCurr[0:T0]=-y[0:To] > A=mpfitfun('myfunct', x,yCurr, ERR, PARINFO=parinfo, WEIGHTS=w, YFIT=yfit, $ BESTNORM=error, /QUIET ) > fitErrors[t]=error > T1values[t]=A[0] > S0values[t]=A[1] > fvalues[t]=A[2] > endfor > min=MIN(fitErrors,idmin) > T1=T1values[idmin] > f=fvalues[idmin] > S0=S0values[idmin] > yfit=myfunct(x,[S0,f,T1]) > plot,x,y > oplot,x,yfit > oplot,x,abs(yfit) > wait,1 > T1map[i,j]=T1 > endfor > endfor > > where x=t;t=t*(findgen(60)+1) dx=dy=256, Im is a 256x256x60 matrix > cheers, > MC First thing would be to remove the "wait,1". This will cancel out 256 * 256 (65536) seconds (which is around 18 hours). Also plotting is time consuming. But there may be additional improvements possible by reorganising the for loops and doing some vectorisation. But this is for the experts here. Cheers, Kai
thank you so much! I'm a new idl user and I'm new in the world of programmers...so I don't know a lot of things. unfortunately for my work is important to view plots to analyze the results! Anyway, I removed "wait,1" for faster work! thanks, Christina
Am 25.11.2013 12:53, schrieb Chrisss: > thank you so much! I'm a new idl user and I'm new in the world of > programmers...so I don't know a lot of things. unfortunately for my > work is important to view plots to analyze the results! Anyway, I > removed "wait,1" for faster work! thanks, Christina If you want to view the plots to analyze the results, one could consider to do calculation first and save the results to disk without plotting. This should be very fast compared to your momentary solution. Afterwards you could load the results and plot them one by one. But anyway, this review is a time consuming job. If you already know what you are looking for in the data, you could search the data automatically. Then you can plot only those datasets you are interested in. As you haven't said anything about the underlying problem, it is not so easy to recommend you further things. If you are looking for eg. outliers or the like you could plot 2d images of your results to find interesting parts. So if you can explain a bit about your data there may possibly be more help to come. Cheers, Kai
Kai, thank you so much!!! ^_^ you were very accurate and I thank you so much for your helpful and your informations. I was not too detailed because I did not want to bore you and take advantage of your precious time, but I really need help! so I try to explain it better: I'm looking for best exponential curve fit, and so the best values about S0, T1 and f (curve parameters) and minimum error. So, I would like to show the plot with the best fit and the min error value in output. I need nested for to display a T1 map of all values calculated within the loops. values refer to everyone of 256x256x60 pixel matrix that I need to built the T1 map
Den måndagen den 25:e november 2013 kl. 15:56:08 UTC+1 skrev Chrisss: > Kai, thank you so much!!! ^_^ > > you were very accurate and I thank you so much for your helpful and your informations. I was not too detailed because I did not want to bore you and take advantage of your precious time, but I really need help! so I try to explain it better: > > I'm looking for best exponential curve fit, and so the best values about S0, T1 and f (curve parameters) and minimum error. So, I would like to show the plot with the best fit and the min error value in output. I need nested for to display a T1 map of all values calculated within the loops. values refer to everyone of 256x256x60 pixel matrix that I need to built the T1 map If I read your code right, your inner loop is used to change the sign of part of the array of data that you are fitting the exponential to. And then you are trying to find the version of your y array that give the best fit. Any reason you can't remove the inner loop and just fit to abs(reform(Im[i,j,*],N_ELEMENTS(x))?
Hi Mats Löfdahl, thanks for your interest! my problem is that my "Im" is a 3D matrix in which every pixel is a curve with 60 elements. Inner loop is for these ones: it does the fitting after the exponential curve inversion. I must follow a scholastic approach that tests all of 60 inversion time T0 within inner loop. I should try a smarter way to find the best inversion time T0. Cheers, Christina
Chrisss writes: Another way you can improve this code is to only calculate unchanging variable once, and outside the loops. For example, you have this, in which you calculate the number of elements of X, which never changes, six times! for i=0, dx-1 do begin for j=0,dy-1 do begin y=reform (Im[i,j,*],N_ELEMENTS(x))) fitErrors=FLTARR(N_ELEMENTS(x)) T1values=FLTARR(N_ELEMENTS(x)) S0values=FLTARR(N_ELEMENTS(x)) fvalues=FLTARR(N_ELEMENTS(x)) for t=0,N_ELEMENTS(x)-1 do begin This can be changed to this: num_x = N_Elements(x) fitErrors=FLTARR(num_x) T1values=FLTARR(num_x) S0values=FLTARR(num_x) fvalues=FLTARR(num_x) for i=0, dx-1 do begin for j=0,dy-1 do begin y=reform (Im[i,j,*],num_x) for t=0,num_x-1 do begin If you need to reinitialize your arrays inside the loop, then do this, rather than re-allocating more memory. I think this is probably why your program crashed after 12 hours. You ran out of dynamic memory on your machine: fitErrors=Temporary(fitErrors)*0.0 T1values=Temporary(T1values)*0.0 S0values=Temporary(S0values)*0.0 fvalues=Temporary(fvalues)*0.0 Cheers, David -- David Fanning, Ph.D. Fanning Software Consulting, Inc. Coyote's Guide to IDL Programming: http://www.idlcoyote.com/ Sepore ma de ni thue. ("Perhaps thou speakest truth.")
Mr Fanning, thanks for your reply! when I calculated fitErrors, T1-s0 and f values within the loops, I wanted to find the best inversion time T0. they do their work outside the loop, too!? wow! I had not really thought! thank you so much cheers, Christina
On 2013-11-25 17:18, Chrisss wrote: > Hi Mats Löfdahl, thanks for your interest! > my problem is that my "Im" is a 3D matrix in which every pixel is a curve with 60 elements. That much I got. :o) > Inner loop is for these ones: it does the fitting after the exponential curve inversion. I must follow a scholastic approach that tests all of 60 inversion time T0 within inner loop. In the loop you do T0=t yCurr=y if (t ne 0) then yCurr[0:T0]=-y[0:To] So you are changing the sign of the first t elements in y. (I'm assuming you don't have To in your program and just happened to write it instead of T0.) What I don't really see is why you'd want to try fitting an exponential to negative numbers, hence the question about just using abs(). Is there some noise that can make the measured (I assume) numbers be negative? I guess I don't know what an "exponential curve inversion" is. Maybe you should make t0 a parameter that you can use in myfunc instead, so instead of testing all possible points where you can flip the sign of your data, you fit your unchanged data to a function that is an exponential where you flip the sign at some point t0, that is part of the fit.
Mr Löfdahl, yes yes, "To" is a write error: To is T0, indeed!I'm sorry! "Im" is a matrix in which there are: noise in background, signal 1 in the middle and signal2 on the inside. The curve that I fit is an exponential with negative and positive numbers; I should flip the negative part but I don't know where is T0 value on x-axis and, above all, procedure that I use imposes to testing all possible t-points within inner for, because I don't know a priori if the best T0 is the first t or the last!
On Monday, November 25, 2013 6:53:26 AM UTC-5, Chrisss wrote: > thank you so much! > > I'm a new idl user and I'm new in the world of programmers...so I don't know a lot of things. unfortunately for my work is important to view plots to analyze the results! Anyway, I removed "wait,1" for faster work! I often put in WAIT statements to make sure that graphics plots are flushed properly to the screen. But you can do wait for a very short amount of time instead of one second, like this, WAIT, 0.0001 and graphics work OK.
I just decided to put out plots from my code or save and show only significative plots thanks for your all considerations and instructions, they were important to me Christina