comp.lang.idl-pvwave archive: archive » Re: Parallel Processing in IDL

Home » Public Forums » archive » Re: Parallel Processing in IDL

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Parallel Processing in IDL [message #72812]

Mon, 11 October 2010 15:13

natha
Messages: 482
Registered: October 2007

Senior Member

On 11 oct, 12:35, Ammar Yusuf <amyus...@gmail.com> wrote:
> What's an easy way to use multiple processors in IDL? I have a large
> program but I want to start with a simple program first. Let's say I
> have an array of a million integers and I have four processors. I want
> to add all the elements in this array and output it. I could split the
> array into four parts and give it to each processor right? Is this
> possible? How would I do this? Thanks!

I wrote a routine to use multi-threading some time ago. I use
IDL_IDLBridge and SHMMAP and it works good for some cases. If you are
interested I can send you a copy.
It only works for functions without output keywords.

Cheers,
nata

Report message to a moderator

Re: Parallel Processing in IDL [message #72813 is a reply to message #72812]

Mon, 11 October 2010 13:01

Michael Galloy
Messages: 1114
Registered: April 2006

Senior Member

On 10/11/10 12:06 PM, Ammar Yusuf wrote:
> Well let's say I have a for loop and inside this for loop I have
> around 50-100 lines and these computations are going out to different
> procedures and functions. Would I still use the IDL Thread Pool?
> Thanks!

The thread pool automatically detects and uses multiple CPUs for vector
operations and particular routines (listed below). So there is no way to
control how work is sent to the various processors. You can control a
few parameters via the CPU routine.

Table 12.2. Thread aware mathematical routines
ACOS ALOG10 ALOG
CONJ COS COSH FINITE FLOOR GAMMA
ASIN ERRORF GAUSSINT ROUND
ATAN EXP IMAGINARY SIN TVSCL
FINDGEN UINDGEN FLOAT
ABS CEIL EXPINT ISHFT
LNGAMMA MATRIX_MULTIPLY PRODUCT SINH SQRT TAN TANH VOIGT

Table 12.3. Thread aware image processing routines
BYTSCL CONVOL FFT

Table 12.4. Thread aware array creation routines
INTERPOLATE DCINDGEN MAKE_ARRAY DOUBLE ULONG
POLY_2D DCOMPLEXARR REPLICATE FIX ULONG64
BINDGEN INDGEN ULINDGEN
BYTARR CINDGEN LINDGEN L64INDGEN
UL64INDGEN

Table 12.5. Thread aware data type conversion routines
BYTE COMPLEX DCOMPLEX LONG LONG64 UINT

Table 12.6. Thread aware array manipulation routines
MAX MIN REPLICATE_INPLACE TOTAL WHERE

Table 12.7. Thread aware programming and IDL control routines
BYTEORDER LOGICAL_AND LOGICAL_OR LOGICAL_TRUE

Mike
--
www.michaelgalloy.com
Research Mathematician
Tech-X Corporation

Report message to a moderator

Re: Parallel Processing in IDL [message #72819 is a reply to message #72813]

Mon, 11 October 2010 11:06

Ammar Yusuf
Messages: 36
Registered: October 2010

Member

On Oct 11, 12:41 pm, mgalloy <mgal...@gmail.com> wrote:
> On 10/11/10 10:35 AM, Ammar Yusuf wrote:
>
>> What's an easy way to use multiple processors in IDL? I have a large
>> program but I want to start with a simple program first. Let's say I
>> have an array of a million integers and I have four processors. I want
>> to add all the elements in this array and output it. I could split the
>> array into four parts and give it to each processor right? Is this
>> possible? How would I do this? Thanks!
>
> IDL's thread pool (search the IDL online help for thread pool for more
> information) will do things like that automatically if you just have a
> system with multiple processors.
>
> If you want to do more complex tasks than just vector operations then
> check out FastDL (it has an MPI interface and a task farming interface,
> depending on the level of control you need):
>
> http://www.txcorp.com/products/FastDL/index.php
>
> Disclosure: I work for Tech-X.
>
> Mike
> --www.michaelgalloy.com
> Research Mathematician
> Tech-X Corporation

Well let's say I have a for loop and inside this for loop I have
around 50-100 lines and these computations are going out to different
procedures and functions. Would I still use the IDL Thread Pool?
Thanks!

Report message to a moderator

Re: Parallel Processing in IDL [message #72820 is a reply to message #72819]

Mon, 11 October 2010 10:51

Haje Korth
Messages: 651
Registered: May 1997

Senior Member

Check the thread pool as Matt suggests. If this does not work for you
you can also check the idl_idlbridge. Haje

On Oct 11, 12:35 pm, Ammar Yusuf <amyus...@gmail.com> wrote:
> What's an easy way to use multiple processors in IDL? I have a large
> program but I want to start with a simple program first. Let's say I
> have an array of a million integers and I have four processors. I want
> to add all the elements in this array and output it. I could split the
> array into four parts and give it to each processor right? Is this
> possible? How would I do this? Thanks!

Report message to a moderator

Re: Parallel Processing in IDL [message #72822 is a reply to message #72820]

Mon, 11 October 2010 09:41

Matt[2]
Messages: 69
Registered: March 2007

Member

On Oct 11, 10:35 am, Ammar Yusuf <amyus...@gmail.com> wrote:
> What's an easy way to use multiple processors in IDL? I have a large
> program but I want to start with a simple program first. Let's say I
> have an array of a million integers and I have four processors. I want
> to add all the elements in this array and output it. I could split the
> array into four parts and give it to each processor right? Is this
> possible? How would I do this? Thanks!

I would look at the documentation about thread pool.

In your case
y = total(array)
will take care of what you want to do assuming your thread pool is set
up properly.

from the total documentation:
"This routine is written to make use of IDL's thread pool, which can
increase execution speed on systems with multiple CPUs. The values
stored in the !CPU system variable control whether IDL uses the thread
pool for a given computation. In addition, you can use the thread pool
keywords TPOOL_MAX_ELTS, TPOOL_MIN_ELTS, and TPOOL_NOTHREAD to
override the defaults established by !CPU for a single invocation of
this routine. See Thread Pool Keywords for details. "

Cheers,
Matt

Report message to a moderator

Re: Parallel Processing in IDL [message #72823 is a reply to message #72822]

Mon, 11 October 2010 09:41

Michael Galloy
Messages: 1114
Registered: April 2006

Senior Member

On 10/11/10 10:35 AM, Ammar Yusuf wrote:
> What's an easy way to use multiple processors in IDL? I have a large
> program but I want to start with a simple program first. Let's say I
> have an array of a million integers and I have four processors. I want
> to add all the elements in this array and output it. I could split the
> array into four parts and give it to each processor right? Is this
> possible? How would I do this? Thanks!

IDL's thread pool (search the IDL online help for thread pool for more
information) will do things like that automatically if you just have a
system with multiple processors.

If you want to do more complex tasks than just vector operations then
check out FastDL (it has an MPI interface and a task farming interface,
depending on the level of control you need):

http://www.txcorp.com/products/FastDL/index.php

Disclosure: I work for Tech-X.

Mike
--
www.michaelgalloy.com
Research Mathematician
Tech-X Corporation

Report message to a moderator

Re: Parallel Processing in IDL [message #72864 is a reply to message #72812]

Wed, 13 October 2010 12:38

Ammar Yusuf
Messages: 36
Registered: October 2010

Member

On Oct 11, 6:13 pm, nata <bernat.puigdomen...@gmail.com> wrote:
> On 11 oct, 12:35, Ammar Yusuf <amyus...@gmail.com> wrote:
>
>> What's an easy way to use multiple processors in IDL? I have a large
>> program but I want to start with a simple program first. Let's say I
>> have an array of a million integers and I have four processors. I want
>> to add all the elements in this array and output it. I could split the
>> array into four parts and give it to each processor right? Is this
>> possible? How would I do this? Thanks!
>
> I wrote a routine to use multi-threading some time ago. I use
> IDL_IDLBridge and SHMMAP and it works good for some cases. If you are
> interested I can send you a copy.
> It only works for functions without output keywords.
>
> Cheers,
> nata

Can you send that code to me please. Thanks!

Report message to a moderator

Re: Parallel Processing in IDL [message #72909 is a reply to message #72812]

Tue, 12 October 2010 05:34

rogass
Messages: 200
Registered: April 2008

Senior Member

On 12 Okt., 00:13, nata <bernat.puigdomen...@gmail.com> wrote:
> On 11 oct, 12:35, Ammar Yusuf <amyus...@gmail.com> wrote:
>
>> What's an easy way to use multiple processors in IDL? I have a large
>> program but I want to start with a simple program first. Let's say I
>> have an array of a million integers and I have four processors. I want
>> to add all the elements in this array and output it. I could split the
>> array into four parts and give it to each processor right? Is this
>> possible? How would I do this? Thanks!
>
> I wrote a routine to use multi-threading some time ago. I use
> IDL_IDLBridge and SHMMAP and it works good for some cases. If you are
> interested I can send you a copy.
> It only works for functions without output keywords.
>
> Cheers,
> nata

Hi Nata,
please send the copy also to me :)

Cheers

CR

Report message to a moderator

Re: Parallel Processing in IDL [message #93159 is a reply to message #72812]

Tue, 03 May 2016 07:18

desilvestri.manuela is currently offline

desilvestri.manuela
Messages: 1
Registered: May 2016

Junior Member

Il giorno martedì 12 ottobre 2010 00:13:48 UTC+2, nata ha scritto:
> On 11 oct, 12:35, Ammar Yusuf <amyus...@gmail.com> wrote:
>> What's an easy way to use multiple processors in IDL? I have a large
>> program but I want to start with a simple program first. Let's say I
>> have an array of a million integers and I have four processors. I want
>> to add all the elements in this array and output it. I could split the
>> array into four parts and give it to each processor right? Is this
>> possible? How would I do this? Thanks!
>
> I wrote a routine to use multi-threading some time ago. I use
> IDL_IDLBridge and SHMMAP and it works good for some cases. If you are
> interested I can send you a copy.
> It only works for functions without output keywords.
>
> Cheers,
> nata

Hi nata!
I'm challenging with IDL multithreading and I have some problems... Can I have a copy of the code you were talking about?

Cheers

MDS

Report message to a moderator

Re: Parallel Processing in IDL [message #94608 is a reply to message #72812]

Sat, 22 July 2017 14:25

charan.harish
Messages: 1
Registered: July 2017

Junior Member

On Tuesday, October 12, 2010 at 3:43:48 AM UTC+5:30, nata wrote:
> On 11 oct, 12:35, Ammar Yusuf <amyus...@gmail.com> wrote:
>> What's an easy way to use multiple processors in IDL? I have a large
>> program but I want to start with a simple program first. Let's say I
>> have an array of a million integers and I have four processors. I want
>> to add all the elements in this array and output it. I could split the
>> array into four parts and give it to each processor right? Is this
>> possible? How would I do this? Thanks!
>
> I wrote a routine to use multi-threading some time ago. I use
> IDL_IDLBridge and SHMMAP and it works good for some cases. If you are
> interested I can send you a copy.
> It only works for functions without output keywords.
>
> Cheers,
> nata

Dear Nata,
Can you send me your script? Further, I would like to tell you my problem that is, I am currently trying to read a bunch of *.sdf files from IDL and then doing post-processing. The problem is since the data size is huge, my IDL script is taking an enormous time.

Hence, can you kindly advise me as of how should I write a parallel IDL script which can read the file in parallel mode and finishes quickly?
Thanking you in advance!

Looking forward to a positive for a positive responce.

Report message to a moderator

Re: Parallel Processing in IDL [message #94609 is a reply to message #94608]

Mon, 24 July 2017 05:56

natha
Messages: 482
Registered: October 2007

Senior Member

Hi,

My library is available online
https://github.com/bernatp3rs/idl_cpu_pm/wiki

Enjoy!

Report message to a moderator

Previous Topic:	move window object programmatically
Next Topic:	Adding more to a Window.Save PDF page

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 15:14:24 PDT 2025

Total time taken to generate the page: 0.00506 seconds