| Re: My code is way to slow.. How can I optimize!? [message #46571] |
Mon, 28 November 2005 16:29 |
leatherback
Messages: 15
Registered: November 2005
|
Junior Member |
|
|
Profiler.. Brilliant!
So, now I know.. 76% of the time is taken up by transposing arrays, and
summing results. I suppose I'll have a look and see whether I can
transpose data outside the loops.
RE David:
Unfortunately: I am now using the routine in some actual data analysis.
So I need the output, and this programm will need to be run on 40
datasets, which would now take a month to run: Optimizing is a must.
Profiling seems to help me focus on the problem area!
RE: Mark Hadfield:
I am so sorry! I'll try to maintain a low-profile on the forum, and be
non-intrusive. I must admit.. My opinion on T-Shirts does not include
IDL shirts. I'd reckon they should give all users a free TShirts,
considering the price they charge for simple licences. Maybe
thinkgeek.com can produce a batch of them? Else..
http://www.thinkgeek.com/tshirts/oreillyshirts/605c/
Anyway.. You've pushed me in the right direction, I am sure,
Thanks!
Jelle.
|
|
|
|
| Re: My code is way to slow.. How can I optimize!? [message #46573 is a reply to message #46571] |
Mon, 28 November 2005 15:57  |
Mark Hadfield
Messages: 783
Registered: May 1995
|
Senior Member |
|
|
jelle.ferwerda@rmit.edu.au wrote:
> I am convinced that my code is sub-sub-sub optimal. Just do not know
> where to start optimizing it, as I am not a programmer, but a mere
> ecologist :-(
A mere ecologist?! How dare you intrude on this forum? This forum is
frequented only by real scientists (ie. physical scientists) who discuss
only serious, high-powered, real science questions like which T-shirts
are the best babe magnets.
I suggest you start by reading the section in the IDL manual on profiling.
--
Mark Hadfield "Kei puwaha te tai nei, Hoea tahi tatou"
m.hadfield@niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)
|
|
|
|
| Re: My code is way to slow.. How can I optimize!? [message #46576 is a reply to message #46573] |
Mon, 28 November 2005 15:43 |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
jelle.ferwerda wrote:
> Anyway. It basically is a series of nested routines, within which I
> perform regressions on subsets, and generate statistics of these
> regressions. Since about 300 lines of the code fall within these loops,
> i am not particularly sure which parts take most of the time. It
> probably is part of an array concenation part, which in a normal
> dataset is performed about 500,000 times. But it could also be the
> regression which is also performed about 500,000 times.
Before you need to know *how* to optimize, you need to know *what* to
optimize. Check out the docs on the PROFILER procedure. Use it to
determine where your program is spending time. Then start asking the
how questions.
> I am convinced that my code is sub-sub-sub optimal. Just do not know
> where to start optimizing it, as I am not a programmer, but a mere
> ecologist :-(
We'll try not to hold that against you. :) Heck, I was a wildlife
biologist once. Hummm. That probably explains a lot about my code.
-Rick
|
|
|
|
| Re: My code is way to slow.. How can I optimize!? [message #46577 is a reply to message #46576] |
Mon, 28 November 2005 15:42 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
jelle.ferwerda@rmit.edu.au writes:
> Anyway. It basically is a series of nested routines, within which I
> perform regressions on subsets, and generate statistics of these
> regressions.
I assume you mean nested loops, which is the usual culprit with
slow routines written in IDL. And you are right, we have (as a group)
had some success speeding this kind of code up:
http://www.dfanning.com/code_tips/slowloops.html
> Since about 300 lines of the code fall within these loops,
> i am not particularly sure which parts take most of the time.
Since it takes a day to run, I can understand not wanting to
run it with timing code. :-)
Still it would be worth while knowing what is slow and what
is REALLY slow.
> It probably is part of an array concenation part, which in a normal
> dataset is performed about 500,000 times. But it could also be the
> regression which is also performed about 500,000 times.
I expect you could find tips for improving the array concatenation
part up by searching through the archives of this newsgroup. If JD
hasn't written about it 50 times it would be a small miracle.
> I am convinced that my code is sub-sub-sub optimal. Just do not know
> where to start optimizing it, as I am not a programmer, but a mere
> ecologist :-(
I'll take your word for its optimized nature. I'm just saying I
don't care to look at 400 lines of code unless I'm also looking at your
credit card details. :-)
But why speed it up? It's served its purpose, it sounds like.
Maybe this is one mess that should just be buried and forgotten.
Let those guys who read your paper figure it out! :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
|
|
|
|
| Re: My code is way to slow.. How can I optimize!? [message #46578 is a reply to message #46577] |
Mon, 28 November 2005 15:27 |
leatherback
Messages: 15
Registered: November 2005
|
Junior Member |
|
|
Hi David,
Thanks for your reply. Nope, i do not have money burning a hole in my
pocket. It is more that I know that the flowchart of the actual routine
I am using will be published in a remote sensing journal soon anyway
(Also already available online as chapter 6 in
http://www.itc.nl/library/Papers_2005/phd/ferwerda.pdf). So I do not
really care about people getting the code I used, It has served my
purpose, and I am now applying it to a dataset, where not this code,
but the data it will be processing is crucial.
Anyway. It basically is a series of nested routines, within which I
perform regressions on subsets, and generate statistics of these
regressions. Since about 300 lines of the code fall within these loops,
i am not particularly sure which parts take most of the time. It
probably is part of an array concenation part, which in a normal
dataset is performed about 500,000 times. But it could also be the
regression which is also performed about 500,000 times.
I am convinced that my code is sub-sub-sub optimal. Just do not know
where to start optimizing it, as I am not a programmer, but a mere
ecologist :-(
J.
|
|
|
|
| Re: My code is way to slow.. How can I optimize!? [message #46581 is a reply to message #46578] |
Mon, 28 November 2005 15:00 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
jelle.ferwerda@rmit.edu.au writes:
> This is my first post here. I was desperately looking for info on the
> web on optimizing IDL code, and came across this group. Looks like you
> might be able to help me..
>
> For the past 3 years I have been working with IDL to get some basic
> processing done for my phd work. Now I have been working on processing
> some signals from a fieldspectrometer, using a bootstrapping routine. I
> was wondering how I can optimize this code, as the current code takes
> over a day for one run to finish.
>
> Is it possible to post some code on this forum? The code is quite long,
> and I'd rather put it as a downloadable file (code is 440 lines).
Posting code implies you have money burning a hole in your
pocket and you wish to spend it. Asking a question about
a particular *aspect* of your code that seems particularly
slow will get you a better response, probably. In any case,
if we need to look at the code we are usually foolish enough
to ask for it. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
