| Re: Processing getting slower and slower? [message #47813] |
Mon, 06 March 2006 09:13 |
Foldy Lajos
Messages: 268
Registered: October 2001
|
Senior Member |
|
|
Hi Zoltan,
I am a linux user, but I think one million files makes the filesystem very
slow (I saw such slowdown on unix machines earlier). Try to create same
directory tree structure, with no more then about 100 files/dirs in a
single directory. With three levels of subdirectories you can have
100*100*100 = one million files with much better access time. This may
help (but the real solution would be using much fewer files).
Also, you can insert 'help, /memory' instructions into the loop to check
memory usage.
regards,
lajos
On Mon, 6 Mar 2006, Zoltan Bartalis wrote:
> Hello all
>
> [Regular reader, first time poster]
>
> I am processing satellite data with IDL, involving reading in data,
> processing it, storing it in several buffer arrays and writing (appending)
> it to lots of binary files (about one million) as soon as any of the buffers
> is full. Without going into more details, a more general question: has
> anyone encountered that while going through several processing cycles
> involving intensive file opening, writing and closing, the processing gets
> slower and slower for each cycle, although the operations in each cycle are
> more or less the same? I am declaring all the buffer arrays and other needed
> variables _before_ looping through the read-in of the input data files, so I
> would expect no change in the memory usage from cycle to cycle. Is this an
> IDL issue or a platform one? Running under Windows XP.
>
> Thanks in advance,
> /Zoltan, Vienna
>
> --
> ------------------------------------------------------------ ------------
> Zoltan Bartalis Institute of Photogrammetry and Remote Sensing
> Vienna University of Technology
> Gusshausstrasse 27-29
> A-1040 Vienna, Austria
> email: zb @ ipf . tuwien . ac . at Tel (++43 1) 58801 12240
> web: http://www.ipf.tuwien.ac.at/zb Fax (++43 1) 58801 12299
> ------------------------------------------------------------ ------------
>
>
>
|
|
|
|
| Re: Processing getting slower and slower? [message #47814 is a reply to message #47813] |
Mon, 06 March 2006 09:02  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Zoltan Bartalis writes:
> I am processing satellite data with IDL, involving reading in data,
> processing it, storing it in several buffer arrays and writing (appending)
> it to lots of binary files (about one million) as soon as any of the buffers
> is full. Without going into more details, a more general question: has
> anyone encountered that while going through several processing cycles
> involving intensive file opening, writing and closing, the processing gets
> slower and slower for each cycle, although the operations in each cycle are
> more or less the same? I am declaring all the buffer arrays and other needed
> variables _before_ looping through the read-in of the input data files, so I
> would expect no change in the memory usage from cycle to cycle. Is this an
> IDL issue or a platform one? Running under Windows XP.
This sounds like a memory-paging-to-disk problem to me.
Are you sure you are destroying/releasing the buffer
memory when you are finished with it?
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
