comp.lang.idl-pvwave archive: archive » Re: How to read first and last line of an ascii file FAST

Home » Public Forums » archive » Re: How to read first and last line of an ascii file FAST

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: How to read first and last line of an ascii file FAST [message #72377 is a reply to message #72335]

Fri, 03 September 2010 02:56

agnarhs
Messages: 4
Registered: May 2009

Junior Member

On 29 Aug, 18:32, Heinz Stege <public.215....@arcor.de> wrote:
> On Sun, 29 Aug 2010 08:32:15 -0700 (PDT), ahs wrote:
>> Hello group,
>
>> My data is stored in a bunch of ascii files, each 5-10MB big. Each
>> line represent a sample with a timestamp. I would like to analyse all
>> my files by reading the timestamp from the first and last line of
>> every file. Getting the timestamp from the last line is the issue
>> here. Right now I'm doing it like
>
>> readf, fid, firstline
>> nlines = FILE_LINES(fnames[i])
>> skip_lun, fid, nlines-2, /LINES
>> readf, fid, lastline
>
>> skip_lun is really slow when the files are big and numerous. Does
>> anyone have a tip of I could do this faster? Is there a way to
>> directly point to the end of the file minus one line?
>
>> regards,
>> Agnar
>
> HiAgnar,
>
> I would expect, that the file_lines function is taking some time too.
> You can try the following:
>
> Get the file size in bytes by the fstat function. Then use point_lun
> to set the file pointer to n bytes before the file end. n should be
> greater than the length of the last line. Read the n bytes into a byte
> array and search for the last occurence of CR and/or LF in that array.
> If you don't find a CR/LF increase n and repeat the steps from
> point_lun to here. Finally convert the bytes after CR/LF to string
> type.
>
> HTH, Heinz

Hi Heinz,

That was a good idea. All the lines in the ascii files are equal with
LINE_B bytes, and doing the following solved my problem.

A = fstat(fid)
readf, fid, firstline
point_lun, fid, A.size-LINE_B
readf, fid, lastline

Agnar

Report message to a moderator

[Message index]

		Re: How to read first and last line of an ascii file FAST By: Jeremy Bailin on Sun, 29 August 2010 18:27
		Re: How to read first and last line of an ascii file FAST By: Heinz Stege on Sun, 29 August 2010 09:32
		Re: How to read first and last line of an ascii file FAST By: agnarhs on Fri, 03 September 2010 02:56
		Re: How to read first and last line of an ascii file FAST By: R.Bauer on Tue, 31 August 2010 09:08

Previous Topic:	about the blackbody radiation curve
Next Topic:	Re: how to change button names on the dialog_message

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Fri Oct 10 04:56:12 PDT 2025

Total time taken to generate the page: 0.80186 seconds