Home »
Public Forums »
archive »
Re: How to improve the efficiency of IDL on cluster
| Re: How to improve the efficiency of IDL on cluster [message #84402 is a reply to message #84395] |
Mon, 03 June 2013 14:25  |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 6/3/13 9:09 AM, Chang Liao wrote:
> Hi! I am currently running IDL on a Linux cluster. Due to the massive
> mount of remote sensing dataset, it takes days to finish just one
> year's calculation. I can specific the computer node,
> memory(48G/96G), and cores(24 cores) for my QBS job. So I am thinking
> shall I use multi-threading or other ways to speedup the
> calculation. Does anyone have any suggestion or experience in this
> related topics, I couldn't find too much on Exelis or Google. I do
> remember IDL has some demo in earlier version, but it's gone in IDL
> 8.0 +.
Do you want a way to use multiple nodes of this cluster or speed up
processing on a single node? Do the nodes have GPUs?
The thread pool is built into IDL and automatically gives some
multi-threading help on certain operations. There is not much you can do
here except tweak some parameters. See the online help for the CPU routine
http://www.exelisvis.com/docs/CPU.html
and the Thread Pool:
http://www.exelisvis.com/docs/The__Thread_Pool.html
We at Tech-X have some third-party solutions to these types of problems:
1. TaskDL is a queue-based task farming solution. You create workers
corresponding to cores on a local or networked machine and then add
tasks to a queue. Workers take new tasks as they finish old ones. This
is for "embarrassingly parallel" problems.
2. mpiDL is a set of wrappers for MPI (Message Passing Interface).
This is more complicated and is for problems which require more
communication between workers.
3. GPULib is an easy way to speed up supported operations using
CUDA-enabled NVIDIA graphics cards.
Contact me for more information about any of these products!
Mike
--
Michael Galloy
www.michaelgalloy.com
Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
Research Mathematician
Tech-X Corporation
|
|
|
|
Current Time: Sun Oct 12 05:11:55 PDT 2025
Total time taken to generate the page: 2.63997 seconds
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
