| Re: Problem when reading many HDF-EOS file [message #49142] |
Tue, 04 July 2006 01:33  |
![Maarten[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Maarten[1] is currently offline")
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
Roni Drori wrote:
> I have an IDL program that reads many HDF-EOS file (using EOS_GD_OPEN
> ....). When I'm reading more the ~100 files the program stops unable to
> open more files.
> What can be done to avoid this problem? (I'm reading simultaneously
> only one file and then closing it with EOS_GD_close)
Others have pointed out that there may be a memory leak in the IDL
routines themselves. Things to try:
- Circumvent the EOS layer, and use the HDF-4 routines directly. If the
leak is in the EOS routines, this might solve your problem.
- The HDF-5 routines do not have this problem, at least not under
Linux. Depending on the volume of the data, h4toh5 may be able to
provide you with a suitable alternative.
I've used the HDF-5 routines to perform approximately 60000 file
open/close operations in a single run without issues under Linux. IDL
6.2 under Windows gave trouble well before that number was reached.
Maarten
|
|
|
|
| Re: Problem when reading many HDF-EOS file [message #49143 is a reply to message #49142] |
Mon, 03 July 2006 23:38  |
peter.albert@gmx.de
Messages: 108
Registered: July 2005
|
Senior Member |
|
|
Hi Roni,
the point of the previous poster about the closing functions is of
course correct, however, I fear there is more to the IDL HDF functions.
Actually, my first posting to this group ever was about a very similar
problem (http://tinyurl.com/ognwa), and since then the only proposed
solution was to wait for the next IDL release. What was strange was
that IDL's memory consumption increased (checked via the Linux "ps"
command), while within IDL a HELP, /MEMORY did not show increasing
values. So I would guess the memory leak is somewhere in the built-in
routines and not necessarily in the user-written IDL code.
Cheers,
Peter
http://tinyurl.com/ognwa
Roni Drori schrieb:
> Hi,
> I have an IDL program that reads many HDF-EOS file (using EOS_GD_OPEN
> ....). When I'm reading more the ~100 files the program stops unable to
> open more files.
> What can be done to avoid this problem? (I'm reading simultaneously
> only one file and then closing it with EOS_GD_close)
>
> Thanks,
> Roni Drori.
|
|
|
|
| Re: Problem when reading many HDF-EOS file [message #49150 is a reply to message #49143] |
Mon, 03 July 2006 07:30 |
James Kuyper
Messages: 425
Registered: March 2000
|
Senior Member |
|
|
Roni Drori wrote:
> Hi,
> I have an IDL program that reads many HDF-EOS file (using EOS_GD_OPEN
> ....). When I'm reading more the ~100 files the program stops unable to
> open more files.
> What can be done to avoid this problem? (I'm reading simultaneously
> only one file and then closing it with EOS_GD_close)
If you never read more than one file at a time, I'd suspect a memory
leak. The HDF-EOS functions contain lots of pairs of book-end
functions: if you open something, you should close it. If you attach
something, you should make sure to detach it, etc.. In general, the
first one of each pair allocates resources (usually including memory),
and the second function must be called to deallocate them; if it isn't,
you have a resource leak. Check all of your calls to such functions,
and make sure that they're matched up properly.
|
|
|
|
| Re: Problem when reading many HDF-EOS file [message #49234 is a reply to message #49150] |
Wed, 05 July 2006 07:19 |
savoie
Messages: 68
Registered: September 1996
|
Member |
|
|
Roni,
I'm tending to agree with kuyper on this one.
kuyper@wizard.net writes:
> Roni Drori wrote:
>> Hi,
>> I have an IDL program that reads many HDF-EOS file (using EOS_GD_OPEN
>> ....). When I'm reading more the ~100 files the program stops unable to
>> open more files.
>> What can be done to avoid this problem? (I'm reading simultaneously
>> only one file and then closing it with EOS_GD_close)
>
> If you never read more than one file at a time, I'd suspect a memory
> leak. The HDF-EOS functions contain lots of pairs of book-end
> functions: if you open something, you should close it. If you attach
> something, you should make sure to detach it, etc.. In general, the
> first one of each pair allocates resources (usually including memory),
> and the second function must be called to deallocate them; if it isn't,
> you have a resource leak. Check all of your calls to such functions,
> and make sure that they're matched up properly.
I had this problem with EOS_GD_ATTACH, in the documentation, there is not a
link to the opposing call EOS_GD_DETACH. Make sure your swath and grid
attatches are detatched before closing the file. Although closing the file
seems like it should clean up the open handles, it doesn't.
Hope this helps.
Matt
--
Matthew Savoie - Scientific Programmer
National Snow and Ice Data Center
(303) 735-0785 http://nsidc.org
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
