comp.lang.idl-pvwave archive: archive » Re: the last line of a large file

Home » Public Forums » archive » Re: the last line of a large file

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: the last line of a large file [message #55333 is a reply to message #55238]

Fri, 10 August 2007 08:00

Carsten Lechte
Messages: 124
Registered: August 2006

Senior Member

Conor wrote:
> lol! Really! What in the world is the point of putting the number of
> lines at the end of the file?

One legitimate reason would be that sometimes you only know how much
data you have until after you have processed it all, especially if the
data sets are so large that you only ever have a small subset in RAM.

A legitimate example are zip archives, where the table of contents is
written to the end of the file, because the the compressed sizes of
the archive members cannot be known in advance, and it would double
the running time to determine the compressed size beforehand, it would
furthermore use twice the disk space to re-write the file with the
contents in front, it would be impossible to keep the whole archive
in RAM before writing it, and finally, one may not be able leave space
for the contents table at the beginning of the file, to be filled in
later, because one would have to know how long the table will be
beforehand...

Of course, this does not mean that the original poster's data has a
legitimate reason for being organised like this.

For the original poster's problem, one idea is to get the file size
in bytes, skip to position file_size-1000, read that small chunk and
parse it for the desired metadata. This might even be faster than
actually counting the lines with FILE_LINES, but it is probably only
worth it if the metadata contains more useful information that just
the number of lines in the file.

chl

Report message to a moderator

[Message index]

		Re: the last line of a large file By: queiny on Fri, 10 August 2007 07:00
		Re: the last line of a large file By: Conor on Fri, 10 August 2007 05:34
		Re: the last line of a large file By: Paul Van Delst[1] on Thu, 09 August 2007 14:22
		Re: the last line of a large file By: Jean H. on Thu, 09 August 2007 13:07
		Re: the last line of a large file By: Carsten Lechte on Fri, 10 August 2007 08:00
		Re: the last line of a large file By: David Fanning on Fri, 10 August 2007 07:10
		Re: the last line of a large file By: Paul Van Delst[1] on Fri, 10 August 2007 07:37

Previous Topic:	Re: Transforming a nonlinear equation
Next Topic:	Removing bad data from an array

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 07:27:09 PDT 2025

Total time taken to generate the page: 1.04222 seconds