| Running a code multiple times automatically [message #92575] |
Sat, 16 January 2016 08:20  |
rojofija
Messages: 9
Registered: January 2015
|
Junior Member |
|
|
Dear friend of the forum.
I am kindly writing to ask you if you know a practical way to run a code several times in a kind of parallel way in IDL.
Lets say I have a code that is called "namecode.pro"
which needs 2 inputs. I have to run it as:
IDL> namecode, 'stringvar1', 'stringvar2'
I would like to do something like
for i=0, (9) do begin
namecode, 'stringvar1', 'stringvar2'
endfor
but without waiting that process in i=0 is over and then the i=1 is over and so on, I just would like to run it 10 times all at the same time just to use 10 processors.
I can open 10 terminals and run the code in each of the terminals, but I would like to do it in a code. It does not have to be with IDL, it could be done in a bash or tcsh as well.
I am trying to use the IDL_IDLBridge option of IDL but with not success at the moment.
Any help is very welcome.
All the best and thanks in advance.
|
|
|
|
| Re: Running a code multiple times automatically [message #92576 is a reply to message #92575] |
Sat, 16 January 2016 13:38  |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 1/16/16 9:20 AM, nd451frd wrote:
> Dear friend of the forum.
>
> I am kindly writing to ask you if you know a practical way to run a
> code several times in a kind of parallel way in IDL.
>
> Lets say I have a code that is called "namecode.pro"
>
> which needs 2 inputs. I have to run it as:
>
> IDL> namecode, 'stringvar1', 'stringvar2'
>
>
> I would like to do something like
>
> for i=0, (9) do begin namecode, 'stringvar1', 'stringvar2' endfor
>
> but without waiting that process in i=0 is over and then the i=1 is
> over and so on, I just would like to run it 10 times all at the same
> time just to use 10 processors.
>
> I can open 10 terminals and run the code in each of the terminals,
> but I would like to do it in a code. It does not have to be with IDL,
> it could be done in a bash or tcsh as well.
>
> I am trying to use the IDL_IDLBridge option of IDL but with not
> success at the moment.
>
> Any help is very welcome.
>
> All the best and thanks in advance.
>
This uses my library at github.com/mgalloy/mglib, particularly the
src/multiprocessing set of routines. These use the IDL_IDLBridge to do
what you are looking to do.
; create 10 processes
pool = mg_pool(n_processes=10)
; I assume the arguments will be different for each process, but here
; we set them all to be the same
str1 = strarr(10) + 'stringvar1'
str2 = strarr(10) + 'stringvar2'
; since NAMECODE is a procedure result will be !null
result = mg_pool->map('namecode', str1, str2, /is_procedure)
; free the processes
obj_destroy, pool
Mike
--
Michael Galloy
www.michaelgalloy.com
Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
|
|
|
|
| Re: Running a code multiple times automatically [message #92577 is a reply to message #92576] |
Sat, 16 January 2016 14:52 |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 1/16/16 2:38 PM, Michael Galloy wrote:
> This uses my library at github.com/mgalloy/mglib, particularly the
> src/multiprocessing set of routines. These use the IDL_IDLBridge to do
> what you are looking to do.
>
> ; create 10 processes
> pool = mg_pool(n_processes=10)
>
> ; I assume the arguments will be different for each process, but here
> ; we set them all to be the same
> str1 = strarr(10) + 'stringvar1'
> str2 = strarr(10) + 'stringvar2'
>
> ; since NAMECODE is a procedure result will be !null
> result = mg_pool->map('namecode', str1, str2, /is_procedure)
>
> ; free the processes
> obj_destroy, pool
>
> Mike
Actually, I simplified this after looking at it:
pool = mg_pool(n_processes=10)
str1 = strarr(10) + 'stringvar1'
str2 = strarr(10) + 'stringvar2'
pool->map, 'namecode', str1, str2
obj_destroy, pool
Mike
--
Michael Galloy
www.michaelgalloy.com
Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
|
|
|
|
| Re: Running a code multiple times automatically [message #92586 is a reply to message #92577] |
Tue, 19 January 2016 00:43 |
rojofija
Messages: 9
Registered: January 2015
|
Junior Member |
|
|
On Saturday, 16 January 2016 22:52:35 UTC, Michael Galloy wrote:
> On 1/16/16 2:38 PM, Michael Galloy wrote:
>> This uses my library at github.com/mgalloy/mglib, particularly the
>> src/multiprocessing set of routines. These use the IDL_IDLBridge to do
>> what you are looking to do.
>>
>> ; create 10 processes
>> pool = mg_pool(n_processes=10)
>>
>> ; I assume the arguments will be different for each process, but here
>> ; we set them all to be the same
>> str1 = strarr(10) + 'stringvar1'
>> str2 = strarr(10) + 'stringvar2'
>>
>> ; since NAMECODE is a procedure result will be !null
>> result = mg_pool->map('namecode', str1, str2, /is_procedure)
>>
>> ; free the processes
>> obj_destroy, pool
>>
>> Mike
>
> Actually, I simplified this after looking at it:
>
> pool = mg_pool(n_processes=10)
> str1 = strarr(10) + 'stringvar1'
> str2 = strarr(10) + 'stringvar2'
> pool->map, 'namecode', str1, str2
> obj_destroy, pool
>
> Mike
> --
> Michael Galloy
> www.michaelgalloy.com
> Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
Dear Michael,
Thank you very much for the detailed solution. For the moment I have not been able to implemented, the cmake at the office is too old so it was not possible to me to compile your C libraries there. I think helpdesk will updated soon.
I tried to do it in my laptop but once I run the cmake following your instructions, it seems to be running and a message appears on the terminal, something like: "The C compiler is GNU 4.x.x" but after that nothing happen.
I was wondering if I am missing something.
|
|
|
|
|
|
| Re: Running a code multiple times automatically [message #92596 is a reply to message #92587] |
Wed, 20 January 2016 01:32 |
rojofija
Messages: 9
Registered: January 2015
|
Junior Member |
|
|
On Tuesday, 19 January 2016 14:09:11 UTC, Mike Galloy wrote:
> On 1/19/16 1:43 am, nd451frd wrote:
>> Thank you very much for the detailed solution. For the moment I have
>> not been able to implemented, the cmake at the office is too old so
>> it was not possible to me to compile your C libraries there. I think
>> helpdesk will updated soon.
>>
>> I tried to do it in my laptop but once I run the cmake following your
>> instructions, it seems to be running and a message appears on the
>> terminal, something like: "The C compiler is GNU 4.x.x" but after
>> that nothing happen.
>>
>> I was wondering if I am missing something.
>>
>
> No need to use cmake. There are some routines in my library that need
> that, but the multiprocessing directory has no compiled C code. Just put
> it in your !path.
>
> Mike
> --
> Michael Galloy
> www.michaelgalloy.com
> Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
Dear Mike,
I implemented your suggestions and it works very well.
I thought it was going to be necessary to use the C code because I got an error in the first tried I did, but after going deeper into the details it was because the IDL in my laptop is so old that does not include the mg_pool.
It works perfectly well in my desktop at the office with IDL 8.4
Thank you very much for your suggestions it is very useful. I have to read a lot now about this multiprocessing strategy which is very interesting.
Following the idea of the multiprocessing, I was wondering if there is a way to delay each of the pools by a small fraction of time or may be start each of then in a sequential mode, for example,
for q=0, (nCPUs-1) do begin
pool[q]->map, 'namecode', str1, str2
endfor
I tried to implement this last part, but it does not work.
All the best and thanks.
|
|
|
|
| Re: Running a code multiple times automatically [message #92633 is a reply to message #92575] |
Wed, 27 January 2016 08:03 |
jiashenyue
Messages: 10
Registered: January 2013
|
Junior Member |
|
|
I am not sure if this is helpful but I was able to throw multiple scripts into the system and let them run by using SPAWN. In order to use this, you simply need to write the command you would like to call in a script file (.txt would be fine) and write a loop to let SPAWN to get it started one by one. Once the loop is done, you will see a bunch of IDL sessions running parallel from the Activity Monitor (in OS X) or in the Task Manager (PC). The only problem is that you CANNOT see the output when you run it. You can only track it by looking at the Activity Monitor or Task Manager to see if your IDL sessions disappear or not.
I guess the following piece of code may help. Sorry I have to use my own code as an example here. It can be easily transferred to your circumstances:
; for p=0, niters-1 do begin
; command='/Applications/exelis/idl85/bin/idl < '+strcompress('script_'+string(p+1,format='(I02)')+'.in',/re move_all)+' &
;
; print, command
;
; spawn, command
; endfor
Here I am using this loop to call multiple scripts (e.g. script_01.in) by creating a command string. If you only want to call ONE function or procedure, there is no need to make a script. Just create the command string with the full directory of IDL executable file and the name of your procedure, following the same way when you run it in a terminal.
The trickiest part is to find the IDL executable file from the directory you got IDL installed. IDL help document provides more details on multiple platforms. I was using OS X and
/Applications/exelis/idl85/bin/idl is where this executable file resides.
Hope this is helpful and good luck!
SJ
On Saturday, January 16, 2016 at 8:20:25 AM UTC-8, nd451frd wrote:
> Dear friend of the forum.
>
> I am kindly writing to ask you if you know a practical way to run a code several times in a kind of parallel way in IDL.
>
> Lets say I have a code that is called "namecode.pro"
>
> which needs 2 inputs. I have to run it as:
>
> IDL> namecode, 'stringvar1', 'stringvar2'
>
>
> I would like to do something like
>
> for i=0, (9) do begin
> namecode, 'stringvar1', 'stringvar2'
> endfor
>
> but without waiting that process in i=0 is over and then the i=1 is over and so on, I just would like to run it 10 times all at the same time just to use 10 processors.
>
> I can open 10 terminals and run the code in each of the terminals, but I would like to do it in a code. It does not have to be with IDL, it could be done in a bash or tcsh as well.
>
> I am trying to use the IDL_IDLBridge option of IDL but with not success at the moment.
>
> Any help is very welcome.
>
> All the best and thanks in advance.
|
|
|
|
| Re: Running a code multiple times automatically [message #92780 is a reply to message #92633] |
Fri, 26 February 2016 09:30 |
rojofija
Messages: 9
Registered: January 2015
|
Junior Member |
|
|
Dear Jiash,
Sorry for the late reply.
I just tested your suggestion. It works very well!!
Thank you very much for helping.
All the best,
nd.
On Wednesday, 27 January 2016 16:03:38 UTC, jiash...@gmail.com wrote:
> I am not sure if this is helpful but I was able to throw multiple scripts into the system and let them run by using SPAWN. In order to use this, you simply need to write the command you would like to call in a script file (.txt would be fine) and write a loop to let SPAWN to get it started one by one. Once the loop is done, you will see a bunch of IDL sessions running parallel from the Activity Monitor (in OS X) or in the Task Manager (PC). The only problem is that you CANNOT see the output when you run it. You can only track it by looking at the Activity Monitor or Task Manager to see if your IDL sessions disappear or not.
>
> I guess the following piece of code may help. Sorry I have to use my own code as an example here. It can be easily transferred to your circumstances:
>
> ; for p=0, niters-1 do begin
> ; command='/Applications/exelis/idl85/bin/idl < '+strcompress('script_'+string(p+1,format='(I02)')+'.in',/re move_all)+' &
> ;
> ; print, command
> ;
> ; spawn, command
> ; endfor
>
> Here I am using this loop to call multiple scripts (e.g. script_01.in) by creating a command string. If you only want to call ONE function or procedure, there is no need to make a script. Just create the command string with the full directory of IDL executable file and the name of your procedure, following the same way when you run it in a terminal.
>
> The trickiest part is to find the IDL executable file from the directory you got IDL installed. IDL help document provides more details on multiple platforms. I was using OS X and
>
> /Applications/exelis/idl85/bin/idl is where this executable file resides.
>
> Hope this is helpful and good luck!
>
> SJ
>
>
> On Saturday, January 16, 2016 at 8:20:25 AM UTC-8, nd451frd wrote:
>> Dear friend of the forum.
>>
>> I am kindly writing to ask you if you know a practical way to run a code several times in a kind of parallel way in IDL.
>>
>> Lets say I have a code that is called "namecode.pro"
>>
>> which needs 2 inputs. I have to run it as:
>>
>> IDL> namecode, 'stringvar1', 'stringvar2'
>>
>>
>> I would like to do something like
>>
>> for i=0, (9) do begin
>> namecode, 'stringvar1', 'stringvar2'
>> endfor
>>
>> but without waiting that process in i=0 is over and then the i=1 is over and so on, I just would like to run it 10 times all at the same time just to use 10 processors.
>>
>> I can open 10 terminals and run the code in each of the terminals, but I would like to do it in a code. It does not have to be with IDL, it could be done in a bash or tcsh as well.
>>
>> I am trying to use the IDL_IDLBridge option of IDL but with not success at the moment.
>>
>> Any help is very welcome.
>>
>> All the best and thanks in advance.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
