| Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85137] |
Tue, 09 July 2013 10:06  |
Chip Helms
Messages: 24
Registered: November 2012
|
Junior Member |
|
|
Hi all,
So I've been playing around with using IDL_IDLBridge to spawn child processes in the hope of reducing the time it takes to get through a pile of data. For testing this method, I've been calculating the average distance from each point on a grid to every other point on that grid. I have a routine that calculates the distance to every point in the grid from a single point. The traditional, single process approach would loop through each grid point and call this function. The child process approach I have written has 4 children, each one executes the function for a different grid point before I advance the loop to the next set of four grid points. The idea is that I have all four children running simultaneously and once all four are complete, I will take the output and concatenate it onto an array. (Once I figure out how to do this I'd like to apply this to performing operations whose results are arrays of data, so having the child process write the results to a file isn't preferable as I'll be processing 32 years worth of 6 hourly data).
So here I come across a couple of questions:
First off, am I crazy for trying to use IDL_IDLBridge outside of anything to do with widgets? All of the examples in the IDL documentation make use of the bridge in conjunction with widgets and, as I've never used widgets before, the example code can be somewhat hard for me to follow. (To this end http://slugidl.pbworks.com/w/page/29199259/Child%20Processes has been very useful).
Second question: What would be the best way to have my loop wait until each child has finished crunching the data? Should I just use a while loop that checks the status of the children every second or so? I feel like there should be a better way than this.
Thanks in advance,
Cheeers,
Chip
P.S. It was really hard not to title this "A question about good parenting with multiple children"
|
|
|
|
|
|
|
|
| Re: Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85149 is a reply to message #85147] |
Tue, 09 July 2013 19:57  |
Russell Ryan
Messages: 122
Registered: May 2012
|
Senior Member |
|
|
On Tuesday, July 9, 2013 5:47:44 PM UTC-4, Chip Helms wrote:
> Thanks, I hadn't seen that link. I'll have to take a closer look tomorrow. I got my actual project working using a while loop to wait for the child processes to finish. I have to say, I'm impressed by the lack of overhead. Using the four child processes resulted in the runtime for each iteration dropping from ~57s to ~15 seconds (my guess is it's because I only create the bridge objects once and then reuse them for each iteration...so the overhead is primarily passing data back and forth).
>
>
>
> Thanks again for the link. By the sound of it, I imagine it's a bit more elegant than what I've labeled as a babysitter loop.
Well, I'd start with mastering the IDL_IDLBridge before integrating it into a widget program.
But before you do that, let me strongly caution you. IDL_IDLBridge has a *MAJOR* bug in it --- there is a serious memory leak. IDL does not free the internal memory usage when an asychronous command. I'm certain of this behavior in Mac OSX, and have heard it's true for Linux (I don't actually have access to a Linux machine, but a friend said it was true). I don't use Windows, but think Exelis said it was not a problem on Windows. This AFFECTS parallelidl as well, because all that code does is all the book keeping of the bridges for you.
As with most things in life, there are work arounds. Unfortunately, this workaround requires you kill IDL and restart, so it's not really a work around per se. The memory leak does happen, but may not really kill you if you're only triggering each bridge a few dozen times. If you're triggering each bridge more than a few 100 times (and it doesn't seem to scale with the data operated on by the bridge), then you can start eating away at an appreciable amount of your RAM. If let run for a long time (many thousand triggers), it will completely kill the computer. The computer will start to swap to disk and the execution will grind to a halt.
Obviously, I've filed a bug report with Exelis. They confirmed everything I've said, and are "working" on it. That was many months (if not close to a year) ago, and it has not been addressed in any of the releases of IDL 8.x.
In short, I highly discourage you from using IDL_IDLBridge until this is fixed --- even if you're writing code that may not be affected (such as Windows) or expect the run to be short (so maybe the memory leaks won't pile up). Eventually, that code (if it's any good) will proliferate to workstations were it will cause problem. In my line of work, I'm aware of two IDL codes that have this problem and I see people happily using them. Normally I wouldn't care, but we share workstations and when they kill the machine, it hurts all of us.
But, if you must. Then yes. You need to poll every bridge and ask for its status. If the status is running, then go to the next bridge. If the bridge is done, then re-trigger. You'll find that a wait of short time between successive polls will actually be faster. Also, you'll need to be very careful because the order in which the bridges are started, may not be the order in which they finish. Consequently, whatever data the bridge processed may need to be sorted out, this can be accomplished by the USERDATA keyword to keep track of which data a bridge processed.
Good luck, you're going to need it.
Russell
|
|
|
|
|
|
| Re: Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85161 is a reply to message #85149] |
Wed, 10 July 2013 09:55 |
natha
Messages: 482
Registered: October 2007
|
Senior Member |
|
|
Hi Russell,
The memory leak you are talking about is it related to the variables defined at the end of the execution?
If you execute something, after getting the result, the variables are not destroyed and they keep using memory. To prevent that I always execute the following commands:
all_var='tmp_var'
IDLBridge->Execute, all_var+'=ROUTINE_INFO(''$MAIN$'',/VARIABLES)'
all_var=self->IDL_IDLBridge::GetVar(all_var)
command='DELVAR, '+STRJOIN(all_var,', ')
IDLBridge->Execute, command
Are you talking about something different?
I guess that using "help, /memory" I would be able to track the usage of memory and see if the bridge definitely has a bug.
nata
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85195 is a reply to message #85194] |
Thu, 11 July 2013 16:20 |
natha
Messages: 482
Registered: October 2007
|
Senior Member |
|
|
On Thursday, July 11, 2013 4:50:45 PM UTC-4, Chip Helms wrote:
> So I suppose ideally, we should add this to the callback routine, right? Is there any way to identify which timer is associated with a given 'execute, /nowait' call?
It seems that the timer ID is concatenated to the other ones. So you just have to get the last ID. It works....
obridge=obj_new('idl_idlbridge')
for i=0,999 do begin
obridge->execute,'a=1',/nowait
while obridge->status() ne 0 do wait,0.0001
timer_id=widget_info(/managed)
ww=where(widget_info(timer_id,/event_pro) eq 'IDL_IDLBRIDGETIMER_EVENT',nn_w)
timer_id=timer_id[ww[nn_w-1]]
widget_control, timer_id, /destroy
print, memory(/high)
endfor
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
