comp.lang.idl-pvwave archive: archive » h5

Home » Public Forums » archive » h5_parse() in the profiler

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: h5_parse() in the profiler [message #93416 is a reply to message #93414]

Fri, 08 July 2016 10:46

Markus Schmassmann
Messages: 129
Registered: April 2016

Senior Member

On 07/08/2016 06:50 PM, Edward Hyer wrote:
> Hi IDL Wizards,
>
> I'm working on an application requiring chunking through a huge
> quantity of HDF5 files. For (EXTREME) ease of coding, my code does
> IDL> H5DATA = H5_PARSE(HDF5_file,/READ_DATA)
> , and then operates on H5DATA. So so easy to code, but that call to
> H5_PARSE() is very time-consuming. I ran the IDL Profiler (as
> elegantly described here:
> http://www.idlcoyote.com/code_tips/whyslow.html), and found that all
> the time was being spent in two routines:
> Routine Calls Only Total
> CREATE_STRUCT (S) 1320 61.130619 0.046311 61.130619
> 0.046311 H5D_READ (S) 92 53.353344 0.579928
> 53.353344 0.579928
>
> The 'H5D_READ' I understand, that is the low-level I/O and it is
> constrained by the system. But the 'CREATE_STRUCT' surprised me.
>
> I guess CREATE_STRUCT() is where the memory allocation is occurring,
> but does it seem right that this takes more time than the actual disk
> I/O?
>
> Any insights are welcome. I could rewrite the code to pull specific
> data out of the HDF5 file by hand, but that would be hundreds of
> lines of code, and I'd really rather not...
>
> --Edward H.
>
create_struct is called much more often, possibly - without looking into
h5d_read - the struct is being created like that:
temp=[]
for i=1,n-1 do temp=create_struct(temp,tagname[i],tagvalue[i])
struct=temp
terribly inefficient, better to create a string and then use
execute(string)

--Markus Schmassmann, IDL wizard apprentice - at best ;-)

Report message to a moderator

[Message index]

		h5_parse() in the profiler By: MarioIncandenza on Fri, 08 July 2016 09:50
		Re: h5_parse() in the profiler By: Markus Schmassmann on Fri, 08 July 2016 10:46
		Re: h5_parse() in the profiler By: MarioIncandenza on Fri, 08 July 2016 21:38
		Re: h5_parse() in the profiler By: Jim Pendleton on Sat, 09 July 2016 07:05

Previous Topic:	keep track of all libraries used
Next Topic:	IDL asynchronous execution fails at >50 child processes

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 19:23:14 PDT 2025

Total time taken to generate the page: 0.00388 seconds