comp.lang.idl-pvwave archive: archive » Re: Multi-core techniques

Home » Public Forums » archive » Re: Multi-core techniques

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Multi-core techniques [message #70424 is a reply to message #70423]

Fri, 16 April 2010 00:35

Maxwell Peck
Messages: 61
Registered: February 2010

Member

On Apr 16, 11:15 am, Tim B <tim.burg...@noaa.gov> wrote:
> I am working with various satellite datasets (e.g. Pathfinder SST)
> where
> most of the prior work has been producing 50km resolution analysis.
> Even
> an intensive piece of code over a full 30-odd year data set could
> still be
> finished overnight. The algorithms that I've seen don't excessively
> use
> array processing (no FFT's or such) and there is a lot of data
> independence i.e.
> values over the course of a few days are used rather than values over
> a complete year.
>
> With a move to higher resolution, i.e. 4km, there is significantly
> more data (approx 150 4km
> pixels in a 50km pixel). Given the data independence, my usual
> approach would be to
> create a thread pool around the number of available cores and
> calculate each data
> 'piece' independently. However IDL doesn't expose threads to the
> programmer. So I'd
> value any thoughts about how to take advantage of multicore CPU's
> (heck, even my laptop
> is a dual core machine). My thoughts are:
>
> - use C(?) to manage forking separate processes that start IDL and
> pass parameters to the
> appropriate procedure to run in IDL
>
> - run a number of IDL programs in parallel, the same code but
> processing different
> temporal regions of the dataset. I can start up IDL in different
> windows on an 8-core
> machine and each seems to be a separate process.
>
> - use a different language/architecture completely :-)
>
> I'd be interested to hear from anyone else trying to take advantage of
> multicore CPU's..
>
> Tim Burgess

I've perhaps misunderstood your post but IDL does have an automatic
thread pool. http://idlastro.gsfc.nasa.gov/idl_html_help/The_IDL_Thread_P ool.html

It certainly seems to work for me on a quad core machine in ENVI. The
trick is making sure the limits on when multicores are used are chosen
properly so it doesn't start paging - I've found this is fairly hard
on Windows 32 bit because of the ridiculous memory management. I
haven't done extensive testing but there is certainly speedup in a lot
of operations when the thread pool is used for ENVI/IDL.

Max

Report message to a moderator

[Message index]

		Re: Multi-core techniques By: Juggernaut on Fri, 16 April 2010 03:49
		Re: Multi-core techniques By: Juggernaut on Fri, 16 April 2010 03:47
		Re: Multi-core techniques By: wita on Fri, 16 April 2010 02:02
		Re: Multi-core techniques By: Tim B on Fri, 16 April 2010 01:19
		Re: Multi-core techniques By: Maxwell Peck on Fri, 16 April 2010 00:35
		Re: Multi-core techniques By: David Fanning on Thu, 15 April 2010 19:14
		Re: Multi-core techniques By: natha on Fri, 16 April 2010 06:54

Previous Topic:	Re: mandelbrot
Next Topic:	mandelbrot

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Tue Dec 02 15:46:29 PST 2025

Total time taken to generate the page: 1.91686 seconds