comp.lang.idl-pvwave archive: archive » Re: Speeding up multiple file reading

Home » Public Forums » archive » Re: Speeding up multiple file reading

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Speeding up multiple file reading [message #47255 is a reply to message #47254]

Thu, 02 February 2006 07:48

Maarten[1]
Messages: 176
Registered: November 2005

Senior Member

clivecook59@gmail.com wrote:

> I have a program where i need to read in multiple files. Currently i
> read in 6000 binary files using a function i have written. This reads
> three columns of data out of each file. To do this i use a loop that
> calls the function to read a file who's data is then added to an array.
> At the moment it takes around 90 seconds to go through the 6000 files.
> Is there any way that i can read this data not using a loop? Or at
> least are there any tips for speeding this up?

Is this data static (a look-up table, for instance?) In that case you
might want to read the data once, and write it back out to a format
that can more easily hold multiple arrays (say, hdf or netcdf). I think
you'll find that hdf is much faster, especially since you don't have to
append to an array, but create it once at the right size.

That said, if you can avoid resizing an array, and just create one
array of the final size and use that instead, I think you'll see some
improvement.

How large are these files? 90 seconds for 6000 files doesn't sound too
shabby, unless each file holds only 3 points. Part of the problem may
be due to overhead of the filesystem itself, which goes avay by using a
single file instead.

Maarten

Report message to a moderator

[Message index]

		Re: Speeding up multiple file reading By: JD Smith on Thu, 02 February 2006 08:56
		Re: Speeding up multiple file reading By: David Fanning on Thu, 02 February 2006 08:50
		Re: Speeding up multiple file reading By: clivecook59 on Thu, 02 February 2006 08:42
		Re: Speeding up multiple file reading By: David Fanning on Thu, 02 February 2006 08:15
		Re: Speeding up multiple file reading By: clivecook59 on Thu, 02 February 2006 08:08
		Re: Speeding up multiple file reading By: Maarten[1] on Thu, 02 February 2006 07:48
		Re: Speeding up multiple file reading By: David Fanning on Thu, 02 February 2006 07:44
		Re: Speeding up multiple file reading By: Paul Van Delst[1] on Thu, 02 February 2006 09:53
		Re: Speeding up multiple file reading By: news.qwest.net on Thu, 02 February 2006 09:51

Previous Topic:	Re: Array slices
Next Topic:	alternative to execute

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 16:59:34 PDT 2025

Total time taken to generate the page: 0.64337 seconds