comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » readcol procedure

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: readcol procedure [message #32157 is a reply to message #32020]

Thu, 12 September 2002 14:05

R.Bauer
Messages: 1424
Registered: November 1998

Senior Member

Liam E. Gumley wrote:

> Reimar Bauer wrote:
> [stuff deleted]
>> if you use the eof method you have to read line by line. As you know idl
>> is an array orientated language so reading in an array is much faster.
>> It's really fast. If you have only 10 lines it doesn't matter but
>> sometimes we got datafiles of nearly 100.000 lines. In this case it is
>> very important.
>
> How much time do you spend in determining the number of lines in the
> file?

Dear Liam,

you are right there was a quite improvement which I missed in the past.
I did the following test to get no problems by internal caches.

I created on my USB Disk V1.1 which allows a max speed of 1MB/s
a file with transpose(sindgen(100000L)) characters.

Then I did a reboot so cache is empty
fileline needs 1.7 seconds to find 100000 lines

After this I rebooted the machine again
(or did someone know how to say linux to clear the filecache)

The following script needs only 1.63 seconds. So it's faster!!
(May be the difference comes from compiling two routines, fileline,filesize)

pro tr
openr,lun,'t1.txt',/get_lun
Z=''
count=0L
while not eof(lun) do begin
readf,lun,z
count=count+1
endwhile
print,count
end

I don't experimented if READS in addition to convert string to values
will need more time as reading again into rows and columns.
I believe reads takes more time.

and you are right too I am using the byte array in my read_data_file
routine not only the number of lines which was calculated from fileline.
The optional output is the bytarr.

There is another important routine bytes2strarr which converts the bytarr
back into string. I have choosen this way to read the data only once. To
get the routine faster is to read again the file because then the file is
in cache and conversion could not be faster.

regards

Reimar

>
> Cheers,
> Liam.
> Practical IDL Programming
> http://www.gumley.com/

--
Forschungszentrum Juelich
email: R.Bauer@fz-juelich.de
http://www.fz-juelich.de/icg/icg-i/
============================================================ ======
a IDL library at ForschungsZentrum Juelich
http://www.fz-juelich.de/icg/icg-i/idl_icglib/idl_lib_intro. html

Report message to a moderator

[Message index]

		readcol procedure By: shih I Chun on Tue, 10 September 2002 10:15
		Re: readcol procedure By: Liam E. Gumley on Thu, 12 September 2002 08:43
		Re: readcol procedure By: R.Bauer on Thu, 12 September 2002 11:39
		Re: readcol procedure By: David Fanning on Thu, 12 September 2002 08:59
		Re: readcol procedure By: R.Bauer on Wed, 11 September 2002 23:42
		Re: readcol procedure By: David Fanning on Wed, 11 September 2002 14:12
		Re: readcol procedure By: Wayne Landsman on Wed, 11 September 2002 07:54
		Re: readcol procedure By: R.Bauer on Tue, 10 September 2002 23:33
		Re: readcol procedure By: Don J Lindler on Tue, 10 September 2002 13:30
		Re: readcol procedure By: R.Bauer on Sun, 15 September 2002 08:16
		Re: readcol procedure By: Pavel A. Romashkin on Fri, 13 September 2002 12:30
		Re: readcol procedure By: R.Bauer on Fri, 13 September 2002 10:43
		Re: readcol procedure By: Pavel A. Romashkin on Fri, 13 September 2002 09:36
		Re: readcol procedure By: David Fanning on Fri, 13 September 2002 08:50
		Re: readcol procedure By: Pavel A. Romashkin on Fri, 13 September 2002 08:39
		Re: readcol procedure By: R.Bauer on Thu, 12 September 2002 14:05
		Re: readcol procedure By: Liam E. Gumley on Thu, 12 September 2002 12:11

Previous Topic:	duplicate .pro filenames
Next Topic:	Re: duplicate .pro filenames

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 11:41:27 PDT 2025

Total time taken to generate the page: 0.00510 seconds