comp.lang.idl-pvwave archive: archive » Re: array concatenation and optimization

Home » Public Forums » archive » Re: array concatenation and optimization

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: array concatenation and optimization [message #26791 is a reply to message #26790]

Wed, 26 September 2001 12:59

Paul van Delst
Messages: 364
Registered: March 1997

Senior Member

Sean Raffuse wrote:
>
> Hello.
>
> I am trying to read a bunch of data from a file to a structure array. I'm
> not sure many data entries the file will have until I have read it and so I
> am increasing the size of the structure array after reading each line.

Count the number of lines first, then create your arrays the required size. (I'm assuming your
file is ASCII data).

> This is working but it has increased the processing time of my loop by an
> order of magnitude. Is there a better way to do this? Is there a reason
> this is so slow?

A while back someone else in this newsgroup more knowledgable than me (I think Craig?)
explained why this is a bad method when the number of points is very large. Suffice it to say
for something like up to maybe a couple hundred points, concatenation is o.k. Any more and
you'd be needlessly sucking up CPU cycles shifting/concatenating large blocks of data on the
fly.

I changed some of my IDL code that reads a huge binary file from the concat method to
estimating the number of points, added some slop and then read it all in a chunk (or in your
case, a line) at a time, counting the points as I went. Before returning I simply truncate the
array. Sped up my code by at least an order of magnitude.

paulv

--
Paul van Delst Religious and cultural
CIMSS @ NOAA/NCEP purity is a fundamentalist
Ph: (301)763-8000 x7274 fantasy
Fax:(301)763-8545 V.S.Naipaul

Report message to a moderator

[Message index]

		Re: array concatenation and optimization By: R.Bauer on Thu, 27 September 2001 00:25
		Re: array concatenation and optimization By: alt on Wed, 26 September 2001 22:05
		Re: array concatenation and optimization By: Andrew Cool on Wed, 26 September 2001 16:49
		Re: array concatenation and optimization By: Pavel A. Romashkin on Wed, 26 September 2001 14:51
		Re: array concatenation and optimization By: Brian Jackel on Wed, 26 September 2001 14:29
		Re: array concatenation and optimization By: Paul van Delst on Wed, 26 September 2001 12:59
		Re: array concatenation and optimization By: R.G.S. on Wed, 26 September 2001 12:22
		Re: array concatenation and optimization By: Craig Markwardt on Thu, 27 September 2001 09:34

Previous Topic:	Re: IDL 5.4 and win2000 problems
Next Topic:	IDL for Windows and MAC...

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Sun Oct 12 04:39:21 PDT 2025

Total time taken to generate the page: 0.08032 seconds