comp.lang.idl-pvwave archive: archive » Re: Reading files with unknown amount of data

Home » Public Forums » archive » Re: Reading files with unknown amount of data

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Reading files with unknown amount of data [message #1354]

Thu, 04 November 1993 09:46

pendleton
Messages: 17
Registered: February 1992

Junior Member

In article <ALANS.93Nov4110936@fallout.juliet.ll.mit.edu>, alans@ll.mit.edu (A.J.Stein) writes:
> I always thought the best approach to the "growing array" problem was to
> "cache" the data in an array of a KByte or so. When the "cache" is full,
> append to array. So, rewriting the previously posted example program this
> way yields:
>
> [example function deleted]
>
> Anyway, if there are better ways to do this, I'd sure love to hear about
> them.
> --
> Alan J.Stein MIT/Lincoln Laboratory alans@LL.mit.edu

"Better" depends a lot on what your needs are, of course. For the particular
data sets we deal with, reading to EOF while counting records, then creating
exact-size arrays and re-reading the file makes the most sense. (Actually,
it's a little more complicated since we use indexed files, but you get the
idea.)

The excess I/O time we incur is siginificantly less than the time it takes the
pager to go out and find more continguous memory for each append. This, after
all, is more I/O unless your data set is small enough to fit in physical
memory.

This method also leaves around a lot more contiguous memory that will still be
available later on.

You should try it both ways, but as your data set size increases, the two-pass
method will, I predict, increase your efficiency in both speed and memory
utilization.

For most of our analysis tasks, we've tried to avoid array appends completely,
even with 400K block (VMS) pagefiles. Appends are easy to code, but they can
become real hogs in just a few passes through a WHILE loop.

Jim Pendleton, Programmer Analyst/Technical Services Specialist
GRO/OSSE, Dept. Physics & Astronomy
Northwestern University
j-pendleton@nwu.edu (708) 491-2748 (708) 491-3135 [FAX]

Report message to a moderator

[Message index]

		Re: Reading files with unknown amount of data By: pendleton on Thu, 04 November 1993 09:46
		Re: Reading files with unknown amount of data By: alans on Thu, 04 November 1993 08:09
		Re: Reading files with unknown amount of data By: zawodny on Thu, 04 November 1993 04:44
		Re: Reading files with unknown amount of data By: brau1231 on Thu, 04 November 1993 03:12
		Re: Reading files with unknown amount of data By: sterne on Thu, 30 September 1993 16:38
		Re: Reading files with unknown amount of data By: zawodny on Thu, 30 September 1993 12:27
		Re: Reading files with unknown amount of data By: pat on Thu, 30 September 1993 10:32
		Re: Reading files with unknown amount of data By: jim on Thu, 30 September 1993 10:47
		Re: Reading files with unknown amount of dat By: oet on Fri, 05 November 1993 00:28

Previous Topic:	Re: Calling External programm
Next Topic:	Re: Reading files with unknown amount of dat

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 09:37:25 PDT 2025

Total time taken to generate the page: 0.32136 seconds