| How to improve the efficiency of IDL on cluster [message #84411] |
Mon, 03 June 2013 08:09  |
Chang Liao
Messages: 5
Registered: December 2012
|
Junior Member |
|
|
Hi!
I am currently running IDL on a Linux cluster.
Due to the massive mount of remote sensing dataset, it takes days to finish just one year's calculation.
I can specific the computer node, memory(48G/96G), and cores(24 cores) for my QBS job. So I am thinking shall I use multi-threading or other ways to speedup the calculation.
Does anyone have any suggestion or experience in this related topics, I couldn't find too much on Exelis or Google. I do remember IDL has some demo in earlier version, but it's gone in IDL 8.0 +.
Please help, thanks in advance!
|
|
|
|
| Re: How to improve the efficiency of IDL on cluster [message #84565 is a reply to message #84411] |
Wed, 05 June 2013 08:18  |
victor.cazcarra
Messages: 4
Registered: May 2013
|
Junior Member |
|
|
El lunes, 3 de junio de 2013 17:09:00 UTC+2, Chang Liao escribió:
> Hi!
>
> I am currently running IDL on a Linux cluster.
>
> Due to the massive mount of remote sensing dataset, it takes days to finish just one year's calculation.
>
> I can specific the computer node, memory(48G/96G), and cores(24 cores) for my QBS job. So I am thinking shall I use multi-threading or other ways to speedup the calculation.
>
> Does anyone have any suggestion or experience in this related topics, I couldn't find too much on Exelis or Google. I do remember IDL has some demo in earlier version, but it's gone in IDL 8.0 +.
>
>
>
> Please help, thanks in advance!
Hi!
Did you try to use the object IDL_IDLBridge? ( http://slugidl.pbworks.com/w/page/29199259/Child%20Processes)
It is like a 'fork' in C to create childs to do a big work in parallel. I have done some work to parallelize big loops and it works very well.
regards,
Víctor
|
|
|
|
| Re: How to improve the efficiency of IDL on cluster [message #84568 is a reply to message #84411] |
Tue, 04 June 2013 20:23 |
Matt Francis
Messages: 94
Registered: May 2010
|
Member |
|
|
On Tuesday, 4 June 2013 01:09:00 UTC+10, Chang Liao wrote:
> Hi!
>
> I am currently running IDL on a Linux cluster.
>
> Due to the massive mount of remote sensing dataset, it takes days to finish just one year's calculation.
>
> I can specific the computer node, memory(48G/96G), and cores(24 cores) for my QBS job. So I am thinking shall I use multi-threading or other ways to speedup the calculation.
>
> Does anyone have any suggestion or experience in this related topics, I couldn't find too much on Exelis or Google. I do remember IDL has some demo in earlier version, but it's gone in IDL 8.0 +.
>
>
>
> Please help, thanks in advance!
Maybe it's heresy around these here parts, but in my view if you're putting IDL and cluster in the same sentence you're doing it wrong. IDL is a great Interactive Data Language but it's simply the wrong tool for the job for large parallel batch processing. You can put lipstick on a pig, but there are much better alternatives out there for doing this kind of stuff, both easier to use (for this use case) and faster.
|
|
|
|
| Re: How to improve the efficiency of IDL on cluster [message #84573 is a reply to message #84411] |
Tue, 04 June 2013 14:26 |
Kenneth Bowman
Messages: 86
Registered: November 2006
|
Member |
|
|
On 2013-06-03 15:09:00 +0000, Chang Liao said:
> Hi!
> I am currently running IDL on a Linux cluster.
> Due to the massive mount of remote sensing dataset, it takes days to
> finish just one year's calculation.
> I can specific the computer node, memory(48G/96G), and cores(24 cores)
> for my QBS job. So I am thinking shall I use multi-threading or other
> ways to speedup the calculation.
> Does anyone have any suggestion or experience in this related topics, I
> couldn't find too much on Exelis or Google. I do remember IDL has some
> demo in earlier version, but it's gone in IDL 8.0 +.
>
> Please help, thanks in advance!
You can get some benefit from multi-threading, and you can go to fully
parallel, as Mike Galloy suggests, but first, can you trivially
parallelize the job by splitting it into independent parts. Such as,
process January as one job and February as a separate job?
If so, and memory permitting, you will probably do much better than
multi-threading.
And, of course, always vectorize first, as Craig suggested.
Ken Bowman
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
