comp.lang.idl-pvwave archive: archive » Re: reading and writing very slow

Home » Public Forums » archive » Re: reading and writing very slow

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: reading and writing very slow [message #75049 is a reply to message #75047]

Fri, 11 February 2011 07:27

ben.bighair
Messages: 221
Registered: April 2007

Senior Member

On 2/11/11 10:16 AM, geoff wrote:
> On Feb 11, 2:09 pm, Reimar Bauer<R.Ba...@fz-juelich.de> wrote:
>> Am 11.02.2011 14:12, schrieb geoff:
>>
>>> Hi
>>
>>> I have some 1-2 GB text files (lots of them!), each containing weather
>>> data for many thousands of stations for 1 year (per file). I need to
>>> get the data out of the year files and into files which have all the
>>> data for 1 weather station. It's easy but slow.
>>
>>> I am reading each year file line by line and appending that line to
>>> the filename of the station (which is one of the fields on the line).
>>> (openw.../append, close). Does opening and closing files so many
>>> times have such an overhead? Is there a quicker way?
>>
>> no
>>
>> but reading line by line has.
>>
> only way i know how. variable length ascii unfortunately :(

Hi,

I can't tell from your description if exactly how you are managing the
output process so this might not be all that helpful. If I had to do
this in IDL, I would use something like Mike Galloy's resizeable array
list. See...

http://michaelgalloy.com/2006/04/24/collection-package-mgarr aylist.html

I might use that array list to aggregate all of the data. When the
aggregation is complete I would then dump it all to file at once.

For each input file you can read in all the data at a swipe and then
parse as needed within IDL. That would might be a lot faster than
trying to read in formatted lines one-at-a-time.

Cheers,
Ben

Report message to a moderator

[Message index]

		Re: reading and writing very slow By: R.Bauer on Mon, 14 February 2011 08:29
		Re: reading and writing very slow By: oxfordenergyservices on Fri, 11 February 2011 09:06
		Re: reading and writing very slow By: ben.bighair on Fri, 11 February 2011 07:27
		Re: reading and writing very slow By: oxfordenergyservices on Fri, 11 February 2011 07:16
		Re: reading and writing very slow By: R.Bauer on Fri, 11 February 2011 06:09

Previous Topic:	Re: GPULIB and ENVI 4.8
Next Topic:	Domain Name Change and Perl Programmer Needed

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 00:05:15 PDT 2025

Total time taken to generate the page: 0.96148 seconds