comp.lang.idl-pvwave archive: archive » Re: array concatenation and optimization

Home » Public Forums » archive » Re: array concatenation and optimization

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: array concatenation and optimization [message #26789 is a reply to message #26784]

Wed, 26 September 2001 14:51

Pavel A. Romashkin
Messages: 531
Registered: November 2000

Senior Member

Paul van Delst wrote:
>
> I changed some of my IDL code that reads a huge binary file from the concat method to
> estimating the number of points, added some slop and then read it all in a chunk (or in your
> case, a line) at a time, counting the points as I went. Before returning I simply truncate the
> array. Sped up my code by at least an order of magnitude.

I have offered this more than once before. It is basically what Paul is
suggesting, except I don't read the number of points. I create a buffer
(array) with 1000-5000 rows (with any number of columns), then read in a
loop, appending the buffer to Result. When READF, Unit, Buffer passes
EOF (and causes IO_ERROR, of course, trying to read past the EOF) I
redimension the buffer only once, using the value from
(FSTATS(unit)).TRANSFER_COUNT. Then, I return to the beginning of last
read and read the last buffer correctly.
In fact, you could even ignore redimensioning, close the file and check
validity of Buffer and truncate it instead, since Buffer does by then
have all data in it from the READF that caused the IO_ERROR.
This eliminates unnecessary reads from the file, as it gets read only
once. You don't need to know the number of rows in it.
I never notice this procedure when it reads data on the fly, it is fast.
Although I don't seem to work on ASCII files that are over 50 Mb.
In case of numeric data mixed with strings, I found no faster way than
to read it all into STRARR and then deaal with it - it is faster to
process string array located in RAM than trying to read incrementally
from the hard drive. Disk I/O is about the slowest thing you computer
can be doing, except maybe accessing a CD-ROM.
Cheers,
Pavel

Report message to a moderator

[Message index]

		Re: array concatenation and optimization By: R.Bauer on Thu, 27 September 2001 00:25
		Re: array concatenation and optimization By: alt on Wed, 26 September 2001 22:05
		Re: array concatenation and optimization By: Andrew Cool on Wed, 26 September 2001 16:49
		Re: array concatenation and optimization By: Pavel A. Romashkin on Wed, 26 September 2001 14:51
		Re: array concatenation and optimization By: Brian Jackel on Wed, 26 September 2001 14:29
		Re: array concatenation and optimization By: Paul van Delst on Wed, 26 September 2001 12:59
		Re: array concatenation and optimization By: R.G.S. on Wed, 26 September 2001 12:22
		Re: array concatenation and optimization By: Craig Markwardt on Thu, 27 September 2001 09:34

Previous Topic:	Re: IDL 5.4 and win2000 problems
Next Topic:	IDL for Windows and MAC...

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sat Oct 11 08:20:52 PDT 2025

Total time taken to generate the page: 2.15936 seconds