| Short array memory allocation [message #61225] |
Thu, 10 July 2008 13:44  |
ray.laymon064
Messages: 8
Registered: July 2008
|
Junior Member |
|
|
Hi Everyone,
I searched the posted messages but couldn't find the answer I am
looking for. Sorry if I repeat the same question.
I am dealing with some data in "grib" format. I perform a simple
operation on each file (open/extract single pixel). The array size of
each file is small (450*250), and I don't store any of the variables
at each loop (overwrite so no memory allocation is needed). However,
there are lots of files (~600,000).
I have a 2.5 Ghz dual-core 4GB memory laptop. When I run my script it,
the task manager tells me IDL uses ~ 12% CPU & ~32MB memory (total PC
CPU usage is ~ 16% and total PC memory usage is ~ 450MB)...And the
whole script takes ~1,5 hours.
I know, there are lots of small files, and naturally it takes time but
is there a way I can speed up the processing time?
Thanks,
Ray
|
|
|
|
|
|
|
|
| Re: Short array memory allocation [message #61962 is a reply to message #61225] |
Mon, 11 August 2008 19:09  |
ray.laymon064
Messages: 8
Registered: July 2008
|
Junior Member |
|
|
On Aug 11, 11:22 am, Brian Larsen <balar...@gmail.com> wrote:
>> So my quesiton is: "Is there a commend/way to create folder inside IDL-
>> script"? Or Should I do it
>> through a shell-script..
>
> But of course IDL can create a folder :)
>
> try a help on directory (? directory) there is all sorts of good
> things there. What you want is FILE_MKDIR.
>
> Cheers,
>
> Brian
>
> ------------------------------------------------------------ --------------
> Brian Larsen
> Boston University
> Center for Space Physicshttp://people.bu.edu/balarsen/Home/IDL
Well, thanks for the hint.. and of course works like a charm..
I knew it would be very trivial..
All the time I have been searching the help/google with wrong keywords
(create directory, folder)..
Best,
Ray
|
|
|
|
| Re: Short array memory allocation [message #61965 is a reply to message #61225] |
Mon, 11 August 2008 08:22 |
Brian Larsen
Messages: 270
Registered: June 2006
|
Senior Member |
|
|
> So my quesiton is: "Is there a commend/way to create folder inside IDL-
> script"? Or Should I do it
> through a shell-script..
But of course IDL can create a folder :)
try a help on directory (? directory) there is all sorts of good
things there. What you want is FILE_MKDIR.
Cheers,
Brian
------------------------------------------------------------ --------------
Brian Larsen
Boston University
Center for Space Physics
http://people.bu.edu/balarsen/Home/IDL
|
|
|
|
| Re: Short array memory allocation [message #61966 is a reply to message #61307] |
Mon, 11 August 2008 08:16 |
ray.laymon064
Messages: 8
Registered: July 2008
|
Junior Member |
|
|
On Jul 11, 10:53 am, FÖLDY Lajos <fo...@rmki.kfki.hu> wrote:
> On Fri, 11 Jul 2008, ray.laymon...@gmail.com wrote:
>
>> That was the answer I was afraid to get..
>> I tried opening and closing the files only and it took the ~same
>> time...
>
>> Still, is there any IDL setting that I can change and make my IDL use
>> more sources?
>> In a way IDL to open/read the file faster.
>
> Probably file lookup in directories is the bottleneck here. This can be
> very slow if the directories contain thousand of files. Try to rearrange
> your ~600.000 files in a tree where a single directory contains no more
> than 256 files.
>
> regards,
> lajos
Hi Lajos,
I may need ~2000 folders (in a tree) to distribute my files..
openw, creates a file, but if the folder to be written doesn't exist,
IDL exists with an error..
So my quesiton is: "Is there a commend/way to create folder inside IDL-
script"? Or Should I do it
through a shell-script..
Best Regards,
Ray
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
