comp.lang.idl-pvwave archive: archive » Re: reading and writing very slow

Home » Public Forums » archive » Re: reading and writing very slow

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: reading and writing very slow [message #75047 is a reply to message #75021]

Fri, 11 February 2011 09:06

oxfordenergyservices is currently offline

oxfordenergyservices
Messages: 56
Registered: January 2009

Member

On Feb 11, 3:27 pm, Ben Tupper <ben.bigh...@gmail.com> wrote:
> On 2/11/11 10:16 AM, geoff wrote:
>
>
>
>
>
>
>
>
>
>> On Feb 11, 2:09 pm, Reimar Bauer<R.Ba...@fz-juelich.de> wrote:
>>> Am 11.02.2011 14:12, schrieb geoff:
>
>>>> Hi
>
>>>> I have some 1-2 GB text files (lots of them!), each containing weather
>>>> data for many thousands of stations for 1 year (per file). I need to
>>>> get the data out of the year files and into files which have all the
>>>> data for 1 weather station. It's easy but slow.
>
>>>> I am reading each year file line by line and appending that line to
>>>> the filename of the station (which is one of the fields on the line).
>>>> (openw.../append, close). Does opening and closing files so many
>>>> times have such an overhead? Is there a quicker way?
>
>>> no
>
>>> but reading line by line has.
>
>> only way i know how. variable length ascii unfortunately :(
>
> Hi,
>
> I can't tell from your description if exactly how you are managing the
> output process so this might not be all that helpful. If I had to do
> this in IDL, I would use something like Mike Galloy's resizeable array
> list. See...
>
> http://michaelgalloy.com/2006/04/24/collection-package-mgarr aylist.html
>
> I might use that array list to aggregate all of the data. When the
> aggregation is complete I would then dump it all to file at once.
>
> For each input file you can read in all the data at a swipe and then
> parse as needed within IDL. That would might be a lot faster than
> trying to read in formatted lines one-at-a-time.

Thanks for this I'll take a look. Turns out is was my fault.
Processing on one linux machine, filesystem on another.

Report message to a moderator

[Message index]

		Re: reading and writing very slow By: R.Bauer on Mon, 14 February 2011 08:29
		Re: reading and writing very slow By: oxfordenergyservices on Fri, 11 February 2011 09:06
		Re: reading and writing very slow By: ben.bighair on Fri, 11 February 2011 07:27
		Re: reading and writing very slow By: oxfordenergyservices on Fri, 11 February 2011 07:16
		Re: reading and writing very slow By: R.Bauer on Fri, 11 February 2011 06:09

Previous Topic:	Re: GPULIB and ENVI 4.8
Next Topic:	Domain Name Change and Perl Programmer Needed

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 15:48:17 PDT 2025

Total time taken to generate the page: 1.84080 seconds