comp.lang.idl-pvwave archive: archive » Reading netcdf/hdf5 files slows down

Home » Public Forums » archive » Reading netcdf/hdf5 files slows down

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Reading netcdf/hdf5 files slows down [message #92026 is a reply to message #92023]

Thu, 01 October 2015 13:45

Paul Van Delst[1]
Messages: 1157
Registered: April 2002

Senior Member

Hello,

On 10/01/15 15:12, KH wrote:
> Hello,
>
> I have several thousand netcdf and hdf5 files that I need to open
> andextract data from. As the program loops through the files, it
> progressively takes longer and longer to read each file. Over the course
> of 24 hours, it went from taking ~6 seconds to completely read the file
> to more than a minute. Since seconds add up when reading 50,000+ files,
> I was hoping someone might know why the program would be slowing down
> and if there is anything (other than restarting IDL) I can do to fix it.
>
> For the netcdf files I'm using the following procedures:
> NCDF_OPEN
> NCDF_INQUIRE
> NCDF_ATTNAME
> NCDF_ATTGET
> NCDF_CLOSE
>
> And for the HDF5:
> H5F_OPEN
> H5G_OPEN
> H5D_OPEN
> H5D_READ
> H5D_CLOSE
> H5G_CLOSE
> H5F_CLOSE
>
> Lastly, I am being sure to close every file once I am done with it
> andremoving any unneeded variables from memory.

Pawing through the HDF5 docs I see there is also a H5S_CLOSE procedure:
http://www.exelisvis.com/docs/h5s_close.html
where it states:
"Failure to release a dataspace using this procedure will result in
resource leaks."

Do you create dataspaces and not release them?

cheers,

paulv

Report message to a moderator

[Message index]

		Reading netcdf/hdf5 files slows down By: khyde on Thu, 01 October 2015 12:12
		Re: Reading netcdf/hdf5 files slows down By: Paul Van Delst[1] on Thu, 01 October 2015 13:45
		Re: Reading netcdf/hdf5 files slows down By: khyde on Fri, 02 October 2015 05:49

Previous Topic:	Attempt to call undefined procedure: 'DEFAULT' error
Next Topic:	How do you using the IDL to add a straight line of 45 degrees inclination to scatter plot of the square?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 13:51:57 PDT 2025

Total time taken to generate the page: 0.00378 seconds