| Re: Speeding up data crunching using IDL_IDLBridge with asychronous execution [message #85167 is a reply to message #85166] |
Wed, 10 July 2013 11:01   |
Russell Ryan
Messages: 122
Registered: May 2012
|
Senior Member |
|
|
BTW, I meant to say. I didn't find this problem. I (re-)discovered it for myself; it had been puzzled over on this blog back in 2010. When I came aware of the problem, I posted to that (then dead) blog
https://groups.google.com/forum/#!searchin/comp.lang.idl-pvw ave/IDL_IDLBridge$20memory/comp.lang.idl-pvwave/5ehCtUB-CCY/ 2Ytp2ulnFAYJ
From this you can see two things:
(1) it seems to be independent of the IDL version. Again, I tested both Mac OSX and Linux, but Exelis tells me that Windows is unaffected (to some degree or another). I'm still running 8.1, but I've poured over "new features" of later releases and never have I seen a mention of the IDL_IDLBridge. Also, I have not tried any of the other IDL_XBridges.
(2) I posted in Jan, which is about the time I pinged Exelis. So, it's been ~6 months (perhaps more if I misremembered something) and still no answer.
On Wednesday, July 10, 2013 1:53:28 PM UTC-4, rr...@stsci.edu wrote:
> On Wednesday, July 10, 2013 1:40:16 PM UTC-4, nata wrote:
>
>> Wow, this is a MAJOR bug! now I see what you mean....
>
>>
>
>>
>
>>
>
>> I've spent a lot time developing a library to allow parallel computing under IDL and now I realize that there is a bug that causes memory problems. f**k !
>
>>
>
>> Thank you for pointing it out. What is the bug report error and how can we know if in a future version this will be solved?
>
>>
>
>>
>
>>
>
>> Thank you Russell to share this,
>
>>
>
>> nata
>
>
>
> @Nata,
>
>
>
> Trust me, I feel your pain. I had written two large codes for analyzing galaxies, one of the codes was an advanced MCMC sampler which utilized the bridges to distribute the processing. This code was very general and (in part) modeled after MPFIT, in terms of usage, flexibility, and organizational philosophy. The unbridged version is fine, but it's just very slow (what computers do not have multiple processors, we should be using them). This problem is perfectly parallelizable, since it has running several simple MCMC chains w/o communicating between the chains, and at some prescribed time having them exchange information. The plan was, run each chain on a separate bridge for some time. Then, bring them together for the information exchange. Then repeat for some amount of time. (It was the Evolutionary MCMC and parallel tempering sampler). Since this was one of primary usages of the bridges, the scripting solution of exiting and restarting IDL is particularly clunky for distributing this MCMC sampler.
>
>
>
> Again, the thing that kills me the most, is when people use the bridges on shared computers --- despite numerous warnings from me. I say, spread the word. Exelis is aware of the issue, but my gut tells me that the more people who complain the faster it gets solved. Outside of my work and your work, I know of 4 codes used in astronomy which suffer from this problem but the authors and users are not aware (or don't care).
>
>
>
> R
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
