| IDL_IDLBridge and the virtual machine [message #83528] |
Wed, 13 March 2013 04:58  |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
Hi,
I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
This is quite annoying, but is there another functional way to parallelize computations in IDL?
I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
Res1 = ComputationFunction(RefImg=0, Img=1)
Res2 = ComputationFunction(RefImg=1, Img=2)
Res3 = ComputationFunction(RefImg=2, Img=3)
...
and put all the results in an array.
Any suggestions?
I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
Thanks,
Helder
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83607 is a reply to message #83528] |
Mon, 18 March 2013 03:17  |
Yngvar Larsen
Messages: 134
Registered: January 2010
|
Senior Member |
|
|
On Friday, 15 March 2013 22:36:21 UTC+1, Mark Piper wrote:
> but I agree, we need a generalized technique for multithreading IDL pro code. I should be careful
> because this is somewhat distant, but we do have a parallel processing API under development
> that we'd like to introduce in IDL 8.4. (Note that 8.3 is scheduled for this fall.)
> Email me if you'd like to be a beta tester; same goes for anyone reading this.
I'll also volunteer as a beta tester.
Regarding bugs in IDL_IDLBRIDGE:
A year ago, I reported an error in this newsgroup ("idl_idlbridge weirdness on unix systems", 2012-02-28): Using more than 15 bridges simultaneously on a linux 64-bit system causes IDL to hang if the bridges are not destroyed in the exact opposite order to the order they were created. You replied that this is a known bug (CR64611), and that Tech Support had identified a possible workaround. However, the bug is still there in IDL 8.2.2 (linux 64-bit), a year later.
--
Yngvar
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83613 is a reply to message #83528] |
Sun, 17 March 2013 07:28 |
markb77
Messages: 217
Registered: July 2006
|
Senior Member |
|
|
On Mar 15, 10:36 pm, Mark Piper <mpi...@ittvis.com> wrote:
> On Thursday, March 14, 2013 5:59:10 PM UTC-6, superchromix wrote:
>
>> ... What I want to do is use IDL to run multi-
>> threaded analysis on a multi-core CPU. I am running cpu-intensive
>> analysis algorithms within stand-alone IDL applications which run in
>> the VM, and I need a way to make full use of the processing power of
>> the CPU.
>
> Just to be clear, IDL's math operations are multithreaded:
>
> http://www.exelisvis.com/docs/Routines_that_Use_the_Th.html
>
> but I agree, we need a generalized technique for multithreading IDL pro code. I should be careful because this is somewhat distant, but we do have a parallel processing API under development that we'd like to introduce in IDL 8.4. (Note that 8.3 is scheduled for this fall.) Email me if you'd like to be a beta tester; same goes for anyone reading this.
>
> We're also looking at better ways to distribute IDL applications. The VM has been OK, but it's time for something better.
>
> mp
hi Mark,
I would also be interested in beta-testing the parallel api, and
anything related to application distribution.
thanks,
Mark
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83614 is a reply to message #83528] |
Sat, 16 March 2013 22:34 |
Russell Ryan
Messages: 122
Registered: May 2012
|
Senior Member |
|
|
On Friday, March 15, 2013 5:36:21 PM UTC-4, Mark Piper wrote:
> On Thursday, March 14, 2013 5:59:10 PM UTC-6, superchromix wrote:
>
>>
>
>> ... What I want to do is use IDL to run multi-
>
>> threaded analysis on a multi-core CPU. I am running cpu-intensive
>
>> analysis algorithms within stand-alone IDL applications which run in
>
>> the VM, and I need a way to make full use of the processing power of
>
>> the CPU.
>
>>
>
>
>
> Just to be clear, IDL's math operations are multithreaded:
>
>
>
> http://www.exelisvis.com/docs/Routines_that_Use_the_Th.html
>
>
>
> but I agree, we need a generalized technique for multithreading IDL pro code. I should be careful because this is somewhat distant, but we do have a parallel processing API under development that we'd like to introduce in IDL 8.4. (Note that 8.3 is scheduled for this fall.) Email me if you'd like to be a beta tester; same goes for anyone reading this.
>
>
>
> We're also looking at better ways to distribute IDL applications. The VM has been OK, but it's time for something better.
>
>
>
> mp
Hi Mark,
I'd love to beta test such a code. The memory leak bug in the IDL_IDLBridge using the /nowait feature is killing me, and love to learn that there is a new/better way to do this explicitly...
-Russell
|
|
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83631 is a reply to message #83528] |
Fri, 15 March 2013 11:38 |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 3/14/13 5:59 PM, superchromix wrote:
> On Mar 14, 5:16 pm, Michael Galloy <mgal...@gmail.com> wrote:
>
>>
>> Well, the VM is free, so it might be a lot to expect to be able to
>> compile code from it as well (although the thread pool is available in
>> the VM, so you automatically use multiple cores for supported
>> operations). Also, remember that runtime licenses are cheaper than
>> development licenses and can perform EXECUTE.
>>
>> Mike
>
> hi Mike,
>
> I think you misread my point. I'm not concerned with the VM being
> able to compile code. What I want to do is use IDL to run multi-
> threaded analysis on a multi-core CPU. I am running cpu-intensive
> analysis algorithms within stand-alone IDL applications which run in
> the VM, and I need a way to make full use of the processing power of
> the CPU.
>
> I don't think this is a lot to expect from a modern programming
> language. I purchase IDL licenses for development work, using the
> full IDE, etc, and on-the-fly data analysis. The stand-alone
> applications I write are freely distributed to my scientific
> collaborators, and there is no possibility of purchasing a license for
> each and every copy of the distributed stand-alone code. If the free
> distribution of stand-alone applications is no longer part of the IDL
> business model, then I guess I should be switching to matlab or
> python.
I agree that technologies for parallelization on multi-core and GPU are
going to be increasingly important for IDL (single cores aren't getting
any faster). But with their current model, multi-core solutions (besides
the thread pool) bump up against their licensing model for distributing
code. I think the VM shows they want to support free distribution of
applications, but it's unclear to me on how to change the VM
limitations/licensing exactly that would accomplish this. Also, the
average IDL user is probably benefited more by the thread pool since
they don't have to even know it exists to get benefits from multi-core.
Mike
--
Michael Galloy
www.michaelgalloy.com
Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
Research Mathematician
Tech-X Corporation
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83642 is a reply to message #83528] |
Thu, 14 March 2013 16:59 |
markb77
Messages: 217
Registered: July 2006
|
Senior Member |
|
|
On Mar 14, 5:16 pm, Michael Galloy <mgal...@gmail.com> wrote:
>
> Well, the VM is free, so it might be a lot to expect to be able to
> compile code from it as well (although the thread pool is available in
> the VM, so you automatically use multiple cores for supported
> operations). Also, remember that runtime licenses are cheaper than
> development licenses and can perform EXECUTE.
>
> Mike
hi Mike,
I think you misread my point. I'm not concerned with the VM being
able to compile code. What I want to do is use IDL to run multi-
threaded analysis on a multi-core CPU. I am running cpu-intensive
analysis algorithms within stand-alone IDL applications which run in
the VM, and I need a way to make full use of the processing power of
the CPU.
I don't think this is a lot to expect from a modern programming
language. I purchase IDL licenses for development work, using the
full IDE, etc, and on-the-fly data analysis. The stand-alone
applications I write are freely distributed to my scientific
collaborators, and there is no possibility of purchasing a license for
each and every copy of the distributed stand-alone code. If the free
distribution of stand-alone applications is no longer part of the IDL
business model, then I guess I should be switching to matlab or
python.
Mark
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83645 is a reply to message #83528] |
Thu, 14 March 2013 11:56 |
Michael Galloy
Messages: 1114
Registered: April 2006
|
Senior Member |
|
|
On 3/14/13 10:40 AM, Helder wrote:
> On Thursday, March 14, 2013 5:16:57 PM UTC+1, Mike Galloy wrote:
>> On 3/14/13 4:09 AM, superchromix wrote:
>>
>>>
>>
>>> I am also interested in doing this, and as far as I know it's
>>> not
>>
>>> possible.
>>
>>>
>>
>>> I think it's ridiculous that in today's world of multi-core CPUs,
>>> IDL
>>
>>> still has no means of running multithreaded analysis from the
>>> VM.
>>
>>> This is a major handicap to its use in cpu-intensive
>>> applications.
>>
>>>
>>
>>
>>
>> Well, the VM is free, so it might be a lot to expect to be able to
>>
>> compile code from it as well (although the thread pool is available
>> in
>>
>> the VM, so you automatically use multiple cores for supported
>>
>> operations). Also, remember that runtime licenses are cheaper than
>>
>> development licenses and can perform EXECUTE.
>>
>>
>>
>> Mike
>>
>> --
>>
>> Michael Galloy
>>
>> www.michaelgalloy.com
>>
>> Modern IDL: A Guide to IDL Programming
>> (http://modernidl.idldev.com)
>>
>> Research Mathematician
>>
>> Tech-X Corporation
>
> Hi Mike, yes, you're right, I don't expect the VM to execute
> commands. I would simply like to run code in parallel. This is not my
> field, so I don't know if there is an *alternative* at all to using
> an execute command (therefor compiling) to run parallel processes. If
> an alternative is existing, I don't know about it and I'm digging for
> this sort of info. For the moment, I'm quite convinced that it is not
> possible with the current status. Then the next question would be if
> something like this is implementable without the execute command. A
> sort of feature request...
Tech-X (disclaimer: I work for Tech-X) has a product, TaskDL, that
allows this type of task farming scenario. Of course, you still need
runtime licenses to run your tasks, but it does allow you to use
multiple cores, or even multiple machines, to runs your tasks.
Mike
--
Michael Galloy
www.michaelgalloy.com
Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
Research Mathematician
Tech-X Corporation
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83648 is a reply to message #83528] |
Thu, 14 March 2013 10:26 |
Russell[1]
Messages: 101
Registered: August 2011
|
Senior Member |
|
|
Hi Helder,
Not exactly. Suppose in your main program you want to initialize 5 bridges. You will need to set (and maybe get) some data to (from) each of them at startup (at completion). You *could* use the SetVar (GetVar) methods to the IDL_IDLBridge, but as you said it can't be used with VM but also those methods are slow as they duplicate the memory usage. Meaning, if you have a data cube that is 1Gb in size that you pass to five bridges, then you have to duplicate that data five times, now ballooning your total RAM usage to 5 Gb. Perhaps that's not an issue for you, but the VM usage seems to be.
The alternative to GetVar and SetVar is to use the shared memory maps. Here any data you want to pass between children and parent processes you will store in a shared memory map. Then, to use the data from within the bridge, you simply initialize the shared memory map. Here's a simple example:
pro test
cd,cur=cwd & cwd=cwd[0]+'/' ;get the current working directory
data2share=[1.,2.,3]
;create shared memory of the data
dim=size(data2share,/dim)
type=size(data2share,/type)
shmmap,'shareddata',dim=dim,type=type
;we'll need this command to set the data to the bridges, but
;only want to do it one time...
cmd='shmmap,"shareddata",dim='+string(dim,f='(I1)')+',type='+string(type,f='(I2)')
;now populate that shared memory
d=shmvar('shareddata')
d[0]=data2share
n=2 ;number of bridges
bridges=objarr(n) ;save the objects
for i=0,n-1 do begin
;create the bridge
bridges[i]=obj_new('IDL_IDLBridge',out=cwd+string(i,f='(I1)' )+'.log')
;set the data to the bridge
bridges[i]->Execute,cmd
;restore the data within a bridge
bridges[i]->Execute,"thisdata = shmvar('shareddata')"
;print the data within the bridge
bridges[i]->Execute,"print,thisdata"
endfor
;destroy the objects
for i=0,n-1 do obj_destroy,bridges[i]
end
Now, look in the files 0.log, 1.log, etc. Then you could get the data out via a similar prescription.
This has (one big advantage) for me, is that you do not duplicate any data in the children processes. *BUT* it's also considerably faster than SetVar and GetVar (as described by Fanning). I'm not sure if it'll work with the VM stuff or not, because you still have to use the Execute method (which smells an awful lot like the execute procedure).
-Russell
On Thursday, March 14, 2013 11:21:24 AM UTC-4, Helder wrote:
> On Thursday, March 14, 2013 3:30:04 PM UTC+1, rr...@stsci.edu wrote:
>
>> On Wednesday, March 13, 2013 7:58:07 AM UTC-4, Helder wrote:
>
>>
>
>>> Hi,
>
>>
>
>>>
>
>>
>
>>> I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
>
>>
>
>>>
>
>>
>
>>> This is quite annoying, but is there another functional way to parallelize computations in IDL?
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> Res1 = ComputationFunction(RefImg=0, Img=1)
>
>>
>
>>>
>
>>
>
>>> Res2 = ComputationFunction(RefImg=1, Img=2)
>
>>
>
>>>
>
>>
>
>>> Res3 = ComputationFunction(RefImg=2, Img=3)
>
>>
>
>>>
>
>>
>
>>> ...
>
>>
>
>>>
>
>>
>
>>> and put all the results in an array.
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> Any suggestions?
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>>
>
>>
>
>>> Thanks,
>
>>
>
>>>
>
>>
>
>>> Helder
>
>>
>
>>
>
>>
>
>> Hi Helder,
>
>>
>
>>
>
>>
>
>> Two things.
>
>>
>
>>
>
>>
>
>> (1) Look at the shmmap and shmvar routines for passing data. You essentially create a shared memory space and load the data there. Then all the nodes use a single string variable to initialize that memory space within their scope. Fanning has a good tutorial on this:
>
>>
>
>> http://www.idlcoyote.com/code_tips/bridge.html
>
>>
>
>>
>
>>
>
>> (2) I'm not sure how you're planning on using the IDL_IDLBridge. But if you use the /nowait feature (which, frankly is the main reason for using the Bridge in the first place), then there is a major memory leak which affects (it affects Linux and Mac as far as I can tell). If you only call the bridges a few times (say <100), it'll probably be fine --- the leaked memory will be smaller than your RAM so you'll never notice. But, if you call it many times >1000 --- then the run time *PER NODE* will increase very rapidly. I've emailed Exelis about this and I pinged them about it yesterday (I have some simulation code that would really benefit from the Bridges). I'll post more if I hear anything.
>
>>
>
>>
>
>>
>
>> Good luck,
>
>>
>
>> Russell
>
>
>
> Hi Russell,
>
> thanks for the tip.
>
> Let me see if I got your point or not...
>
> What you're trying to tell me is to use shared memory on which different processes share their status and results and the main code launches and waits for these processes to finish running?
>
> This way I would avoid using IDL_IDLBridge, and simply Spawn .sav files that read the info from shared memory.
>
> I can see that this could work, but every time I spawn a new .sav file, the VM will up with the splash screen and each process has to be "clicked on".
>
>
>
> Thanks for the info on the memory allocation sizes (from the Coyote) and the memory leak problems. However, the latter hopefully does not influence me too much because I have to deal with something like hundreds of processes.
>
>
>
> Oh yeah... the idea is that I'm looking for image drift in a stack of n images (up to a few hundreds at a time). Each may take a few seconds to compute and have 4+ processors would speed up things (I hope).
>
>
>
> Cheers,
>
> Helder
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83652 is a reply to message #83528] |
Thu, 14 March 2013 09:40 |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
On Thursday, March 14, 2013 5:16:57 PM UTC+1, Mike Galloy wrote:
> On 3/14/13 4:09 AM, superchromix wrote:
>
>>
>
>> I am also interested in doing this, and as far as I know it's not
>
>> possible.
>
>>
>
>> I think it's ridiculous that in today's world of multi-core CPUs, IDL
>
>> still has no means of running multithreaded analysis from the VM.
>
>> This is a major handicap to its use in cpu-intensive applications.
>
>>
>
>
>
> Well, the VM is free, so it might be a lot to expect to be able to
>
> compile code from it as well (although the thread pool is available in
>
> the VM, so you automatically use multiple cores for supported
>
> operations). Also, remember that runtime licenses are cheaper than
>
> development licenses and can perform EXECUTE.
>
>
>
> Mike
>
> --
>
> Michael Galloy
>
> www.michaelgalloy.com
>
> Modern IDL: A Guide to IDL Programming (http://modernidl.idldev.com)
>
> Research Mathematician
>
> Tech-X Corporation
Hi Mike,
yes, you're right, I don't expect the VM to execute commands. I would simply like to run code in parallel. This is not my field, so I don't know if there is an *alternative* at all to using an execute command (therefor compiling) to run parallel processes. If an alternative is existing, I don't know about it and I'm digging for this sort of info. For the moment, I'm quite convinced that it is not possible with the current status.
Then the next question would be if something like this is implementable without the execute command. A sort of feature request...
Cheers,
Helder
|
|
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83654 is a reply to message #83528] |
Thu, 14 March 2013 08:21 |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
On Thursday, March 14, 2013 3:30:04 PM UTC+1, rr...@stsci.edu wrote:
> On Wednesday, March 13, 2013 7:58:07 AM UTC-4, Helder wrote:
>
>> Hi,
>
>>
>
>> I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
>
>>
>
>> This is quite annoying, but is there another functional way to parallelize computations in IDL?
>
>>
>
>>
>
>>
>
>> I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
>
>>
>
>>
>
>>
>
>> Res1 = ComputationFunction(RefImg=0, Img=1)
>
>>
>
>> Res2 = ComputationFunction(RefImg=1, Img=2)
>
>>
>
>> Res3 = ComputationFunction(RefImg=2, Img=3)
>
>>
>
>> ...
>
>>
>
>> and put all the results in an array.
>
>>
>
>>
>
>>
>
>> Any suggestions?
>
>>
>
>>
>
>>
>
>> I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
>
>>
>
>>
>
>>
>
>> Thanks,
>
>>
>
>> Helder
>
>
>
> Hi Helder,
>
>
>
> Two things.
>
>
>
> (1) Look at the shmmap and shmvar routines for passing data. You essentially create a shared memory space and load the data there. Then all the nodes use a single string variable to initialize that memory space within their scope. Fanning has a good tutorial on this:
>
> http://www.idlcoyote.com/code_tips/bridge.html
>
>
>
> (2) I'm not sure how you're planning on using the IDL_IDLBridge. But if you use the /nowait feature (which, frankly is the main reason for using the Bridge in the first place), then there is a major memory leak which affects (it affects Linux and Mac as far as I can tell). If you only call the bridges a few times (say <100), it'll probably be fine --- the leaked memory will be smaller than your RAM so you'll never notice. But, if you call it many times >1000 --- then the run time *PER NODE* will increase very rapidly. I've emailed Exelis about this and I pinged them about it yesterday (I have some simulation code that would really benefit from the Bridges). I'll post more if I hear anything.
>
>
>
> Good luck,
>
> Russell
Hi Russell,
thanks for the tip.
Let me see if I got your point or not...
What you're trying to tell me is to use shared memory on which different processes share their status and results and the main code launches and waits for these processes to finish running?
This way I would avoid using IDL_IDLBridge, and simply Spawn .sav files that read the info from shared memory.
I can see that this could work, but every time I spawn a new .sav file, the VM will up with the splash screen and each process has to be "clicked on".
Thanks for the info on the memory allocation sizes (from the Coyote) and the memory leak problems. However, the latter hopefully does not influence me too much because I have to deal with something like hundreds of processes.
Oh yeah... the idea is that I'm looking for image drift in a stack of n images (up to a few hundreds at a time). Each may take a few seconds to compute and have 4+ processors would speed up things (I hope).
Cheers,
Helder
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83655 is a reply to message #83528] |
Thu, 14 March 2013 07:30 |
Russell Ryan
Messages: 122
Registered: May 2012
|
Senior Member |
|
|
On Wednesday, March 13, 2013 7:58:07 AM UTC-4, Helder wrote:
> Hi,
>
> I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
>
> This is quite annoying, but is there another functional way to parallelize computations in IDL?
>
>
>
> I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
>
>
>
> Res1 = ComputationFunction(RefImg=0, Img=1)
>
> Res2 = ComputationFunction(RefImg=1, Img=2)
>
> Res3 = ComputationFunction(RefImg=2, Img=3)
>
> ...
>
> and put all the results in an array.
>
>
>
> Any suggestions?
>
>
>
> I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
>
>
>
> Thanks,
>
> Helder
Hi Helder,
Two things.
(1) Look at the shmmap and shmvar routines for passing data. You essentially create a shared memory space and load the data there. Then all the nodes use a single string variable to initialize that memory space within their scope. Fanning has a good tutorial on this:
http://www.idlcoyote.com/code_tips/bridge.html
(2) I'm not sure how you're planning on using the IDL_IDLBridge. But if you use the /nowait feature (which, frankly is the main reason for using the Bridge in the first place), then there is a major memory leak which affects (it affects Linux and Mac as far as I can tell). If you only call the bridges a few times (say <100), it'll probably be fine --- the leaked memory will be smaller than your RAM so you'll never notice. But, if you call it many times >1000 --- then the run time *PER NODE* will increase very rapidly. I've emailed Exelis about this and I pinged them about it yesterday (I have some simulation code that would really benefit from the Bridges). I'll post more if I hear anything.
Good luck,
Russell
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #83656 is a reply to message #83528] |
Thu, 14 March 2013 07:30 |
Russell Ryan
Messages: 122
Registered: May 2012
|
Senior Member |
|
|
On Wednesday, March 13, 2013 7:58:07 AM UTC-4, Helder wrote:
> Hi,
>
> I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
>
> This is quite annoying, but is there another functional way to parallelize computations in IDL?
>
>
>
> I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
>
>
>
> Res1 = ComputationFunction(RefImg=0, Img=1)
>
> Res2 = ComputationFunction(RefImg=1, Img=2)
>
> Res3 = ComputationFunction(RefImg=2, Img=3)
>
> ...
>
> and put all the results in an array.
>
>
>
> Any suggestions?
>
>
>
> I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
>
>
>
> Thanks,
>
> Helder
Hi Helder,
Two things.
(1) Look at the shmmap and shmvar routines for passing data. You essentially create a shared memory space and load the data there. Then all the nodes use a single string variable to initialize that memory space within their scope. Fanning has a good tutorial on this:
http://www.idlcoyote.com/code_tips/bridge.html
(2) I'm not sure how you're planning on using the IDL_IDLBridge. But if you use the /nowait feature (which, frankly is the main reason for using the Bridge in the first place), then there is a major memory leak which affects (it affects Linux and Mac as far as I can tell). If you only call the bridges a few times (say <100), it'll probably be fine --- the leaked memory will be smaller than your RAM so you'll never notice. But, if you call it many times >1000 --- then the run time *PER NODE* will increase very rapidly. I've emailed Exelis about this and I pinged them about it yesterday (I have some simulation code that would really benefit from the Bridges). I'll post more if I hear anything.
Good luck,
Russell
|
|
|
|
|
|
|
|
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #86442 is a reply to message #83607] |
Fri, 08 November 2013 11:36 |
gombgg
Messages: 7
Registered: November 2013
|
Junior Member |
|
|
I just ran into this problem myself on a 24-core system running IDL 8.2.3 on 64 bit Linux. I can create 24 bridge objects and use them for processing, but if I destroy them in the same order they were created then IDL hangs after destroying the 8th one. If I destroy them in reverse order, everything works fine.
On Monday, March 18, 2013 4:17:29 AM UTC-6, Yngvar Larsen wrote:
> On Friday, 15 March 2013 22:36:21 UTC+1, Mark Piper wrote:
>
>
>
>> but I agree, we need a generalized technique for multithreading IDL pro code. I should be careful
>
>> because this is somewhat distant, but we do have a parallel processing API under development
>
>> that we'd like to introduce in IDL 8.4. (Note that 8.3 is scheduled for this fall.)
>
>> Email me if you'd like to be a beta tester; same goes for anyone reading this.
>
>
>
> I'll also volunteer as a beta tester.
>
>
>
> Regarding bugs in IDL_IDLBRIDGE:
>
>
>
> A year ago, I reported an error in this newsgroup ("idl_idlbridge weirdness on unix systems", 2012-02-28): Using more than 15 bridges simultaneously on a linux 64-bit system causes IDL to hang if the bridges are not destroyed in the exact opposite order to the order they were created. You replied that this is a known bug (CR64611), and that Tech Support had identified a possible workaround. However, the bug is still there in IDL 8.2.2 (linux 64-bit), a year later.
>
>
>
> --
>
> Yngvar
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: IDL_IDLBridge and the virtual machine [message #89145 is a reply to message #83528] |
Mon, 28 July 2014 04:36 |
Jie Zhou
Messages: 27
Registered: February 2014
|
Junior Member |
|
|
On Wednesday, March 13, 2013 12:58:07 PM UTC+1, Helder wrote:
> Hi,
>
> I would like to use the IDL_IDLBridge, but I just read in the documentation that it is not possible to use execute, getvar and setvar methods in VM mode.
>
> This is quite annoying, but is there another functional way to parallelize computations in IDL?
>
>
>
> I basically have a list of n images (n~100, maybe more) and I perform computations on images k with k-1 for the whole list. I wanted to start computations separately:
>
>
>
> Res1 = ComputationFunction(RefImg=0, Img=1)
>
> Res2 = ComputationFunction(RefImg=1, Img=2)
>
> Res3 = ComputationFunction(RefImg=2, Img=3)
>
> ...
>
> and put all the results in an array.
>
>
>
> Any suggestions?
>
>
>
> I could try writing a batch process and producing a .sav that puts the result in a file and so on... but this seems a bit "dirty".
>
>
>
> Thanks,
>
> Helder
Just after copy the license folder under the IDLDIR directory to the distribution directory, the multiprocessing CUP works. It seems now the save file works in runtime mode. Good luck.
Jie
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
