| Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85137] |
Tue, 09 July 2013 10:06  |
Chip Helms
Messages: 24
Registered: November 2012
|
Junior Member |
|
|
Hi all,
So I've been playing around with using IDL_IDLBridge to spawn child processes in the hope of reducing the time it takes to get through a pile of data. For testing this method, I've been calculating the average distance from each point on a grid to every other point on that grid. I have a routine that calculates the distance to every point in the grid from a single point. The traditional, single process approach would loop through each grid point and call this function. The child process approach I have written has 4 children, each one executes the function for a different grid point before I advance the loop to the next set of four grid points. The idea is that I have all four children running simultaneously and once all four are complete, I will take the output and concatenate it onto an array. (Once I figure out how to do this I'd like to apply this to performing operations whose results are arrays of data, so having the child process write the results to a file isn't preferable as I'll be processing 32 years worth of 6 hourly data).
So here I come across a couple of questions:
First off, am I crazy for trying to use IDL_IDLBridge outside of anything to do with widgets? All of the examples in the IDL documentation make use of the bridge in conjunction with widgets and, as I've never used widgets before, the example code can be somewhat hard for me to follow. (To this end http://slugidl.pbworks.com/w/page/29199259/Child%20Processes has been very useful).
Second question: What would be the best way to have my loop wait until each child has finished crunching the data? Should I just use a while loop that checks the status of the children every second or so? I feel like there should be a better way than this.
Thanks in advance,
Cheeers,
Chip
P.S. It was really hard not to title this "A question about good parenting with multiple children"
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
