| Re: Processing getting slower and slower? [message #47873 is a reply to message #47815] |
Wed, 08 March 2006 06:49   |
Nigel Wade
Messages: 286
Registered: March 1998
|
Senior Member |
|
|
Zoltan Bartalis wrote:
> Hello all
>
> [Regular reader, first time poster]
>
> I am processing satellite data with IDL, involving reading in data,
> processing it, storing it in several buffer arrays and writing (appending)
> it to lots of binary files (about one million) as soon as any of the buffers
> is full. Without going into more details, a more general question: has
> anyone encountered that while going through several processing cycles
> involving intensive file opening, writing and closing, the processing gets
> slower and slower for each cycle, although the operations in each cycle are
> more or less the same? I am declaring all the buffer arrays and other needed
> variables _before_ looping through the read-in of the input data files, so I
> would expect no change in the memory usage from cycle to cycle. Is this an
> IDL issue or a platform one? Running under Windows XP.
>
> Thanks in advance,
> /Zoltan, Vienna
>
Is there any change in memory usage as the loops progress?
It might be a filesystem issue. As you continually open/append/close the files
they grow in size, and each append takes longer as it has to find the end of a
progressively longer file. I think this will be made much worse by the fact you
are writing many files simultaneously, so each file will become very badly
fragmented, and fragmented files take much longer to locate to EOF.
--
Nigel Wade, System Administrator, Space Plasma Physics Group,
University of Leicester, Leicester, LE1 7RH, UK
E-mail : nmw@ion.le.ac.uk
Phone : +44 (0)116 2523548, Fax : +44 (0)116 2523555
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
