| Memory leak in IDL hdf5 library? [message #74508] |
Tue, 18 January 2011 11:07  |
Eddie Schlafly
Messages: 5
Registered: January 2011
|
Junior Member |
|
|
I think I've found a memory leak in the IDL hdf5 library and I wanted to
know if anyone knows what I'm doing wrong or what is going on. I've
appended some code that leaks memory based on the IDL hdf5 example code.
I modified the example ex_create_hdf5 routine to write a structure
containing an array of structures; if the number of elements in this
array is greater than 1, I see a memory leak.
Running:
ex_create_hdf5 & for i=0l,10 do ex_read_hdf5
causes the amount of memory allocated to IDL according to "top" to
increase by ~700 MB every time the line is run, despite the fact that the
written file is only about 40 KB.
IDL's
help, /memory
does not change and gives no indication that hundreds of MB of memory are
being used by IDL. Likewise,
help, /heap
shows no pointers or objects.
I am running:
help, !version, /st
** Structure !VERSION, 8 tags, length=104, data length=100:
ARCH STRING 'x86_64'
OS STRING 'linux'
OS_FAMILY STRING 'unix'
OS_NAME STRING 'linux'
RELEASE STRING '7.1.1'
BUILD_DATE STRING 'Aug 21 2009'
MEMORY_BITS INT 64
FILE_OFFSET_BITS
INT 64
Any hints as to what is going on?
Thanks for your help,
Eddie Schlafly
--
PRO ex_create_hdf5
file = 'hdf5_test.h5'
fid = H5F_CREATE(file)
;; create data
data0 = { a:replicate({b:0},2) }
data = replicate(data0, 10000)
;; get data type and space, needed to create the dataset
datatype_id = H5T_IDL_CREATE(data)
dataspace_id = H5S_CREATE_SIMPLE(size(data,/DIMENSIONS))
;; create dataset in the output file
dataset_id = H5D_CREATE(fid,$
'Sample data',datatype_id,dataspace_id)
;; write data to dataset
H5D_WRITE,dataset_id,data
;; close all open identifiers
H5D_CLOSE,dataset_id
H5S_CLOSE,dataspace_id
H5T_CLOSE,datatype_id
H5F_CLOSE,fid
END
PRO ex_read_hdf5
; Open the HDF5 file.
file = 'hdf5_test.h5'
file_id = H5F_OPEN(file)
; Open the image dataset within the file.
; This is located within the /images group.
; We could also have used H5G_OPEN to open up the group first.
dataset_id1 = H5D_OPEN(file_id, 'Sample data')
; Read in the actual image data.
image = H5D_READ(dataset_id1)
; Close all our identifiers so we don't leak resources.
H5D_CLOSE, dataset_id1
H5F_CLOSE, file_id
END
|
|
|
|
|
|
|
|
| Re: Memory leak in IDL hdf5 library? [message #74599 is a reply to message #74508] |
Mon, 24 January 2011 13:08  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Eddie Schlafly writes:
> I take it that you don't see any leak under Windows?
No, I don't think so. I don't see anything dramatic
happening with memory.
>
> Thanks for pointing out h5_close! I just followed the example files from
> the hdf5 documentation, which didn't used h5_close.
>
> If I add an h5_close at the end of ex_read_hdf5 and ex_create_hdf5, I no
> longer leak memory. If, however, I just try to h5_close when I'm done
> reading in a bunch of files, I still leak.
>
> Adding h5_close, however, doesn't change the problem I am having where
> the amount of memory used is many times the size of the file being read
> in. The ~40K file I am reading in ends up reading in to about 50 MB of
> memory, which is freed by the h5_close. The files are also slower to
> read in than I would expect. If I write out a structure with a zero in
> it, rather than a nested structure with a zero in it, things read in
> quickly and no memory is leaked, even without h5_close.
I'm really just waving my hands here, but if I read the
H5_CLOSE documentation correctly, I would guess that there
is an enormous amount of overhead in "linking" to the H5
libraries. I would guess it is this memory that is freed
with H5_CLOSE. Reading the files is probably slower because
you have to load and unload all this overhead each time.
Have you discussed this with ITTVIS? What do they recommend?
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Memory leak in IDL hdf5 library? [message #74604 is a reply to message #74508] |
Mon, 24 January 2011 12:50 |
Eddie Schlafly
Messages: 5
Registered: January 2011
|
Junior Member |
|
|
Thanks for your help, David.
David Fanning writes:
> I don't notice any change on my Windows machines with or without this
> extra "close" statement, but I wonder if it would make a difference for
> you?
I take it that you don't see any leak under Windows?
Thanks for pointing out h5_close! I just followed the example files from
the hdf5 documentation, which didn't used h5_close.
If I add an h5_close at the end of ex_read_hdf5 and ex_create_hdf5, I no
longer leak memory. If, however, I just try to h5_close when I'm done
reading in a bunch of files, I still leak.
Adding h5_close, however, doesn't change the problem I am having where
the amount of memory used is many times the size of the file being read
in. The ~40K file I am reading in ends up reading in to about 50 MB of
memory, which is freed by the h5_close. The files are also slower to
read in than I would expect. If I write out a structure with a zero in
it, rather than a nested structure with a zero in it, things read in
quickly and no memory is leaked, even without h5_close.
Thanks a lot,
Eddie Schlafly
|
|
|
|
|
|
|
|
| Re: Memory leak in IDL hdf5 library? [message #74730 is a reply to message #74508] |
Tue, 01 February 2011 02:12 |
Alain Kattnig
Messages: 9
Registered: November 2009
|
Junior Member |
|
|
On 1 fév, 00:43, Eddie Schlafly <schla...@hotmail.com> wrote:
> Alain Kattnig writes:
>> If you are using IDL 7, there is a patch for HDF 5, called idl712patch
>> which settled various troubles for me.
>
> Couldn't get a hold of this, but I talked to ITTVIS and they have filed a
> bug report now. I seem the same behavior under IDL 8 too. I guess I'll
> just try harder to avoid nested structures in hdf5 files in the future.
>
> Thanks everyone,
>
> Eddie Schlafly
Unfortunately the IDL 7.1.2 patch is only available for Windows
systems ...
|
|
|
|
| Re: Memory leak in IDL hdf5 library? [message #74734 is a reply to message #74589] |
Mon, 31 January 2011 15:43 |
Eddie Schlafly
Messages: 5
Registered: January 2011
|
Junior Member |
|
|
Alain Kattnig writes:
> If you are using IDL 7, there is a patch for HDF 5, called idl712patch
> which settled various troubles for me.
Couldn't get a hold of this, but I talked to ITTVIS and they have filed a
bug report now. I seem the same behavior under IDL 8 too. I guess I'll
just try harder to avoid nested structures in hdf5 files in the future.
Thanks everyone,
Eddie Schlafly
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
