| Re: slow file-handling [message #6919 is a reply to message #6913] |
Mon, 02 September 1996 00:00   |
David Theil
Messages: 11
Registered: May 1996
|
Junior Member |
|
|
Jorn Helbert wrote:
>
> Hi,
> I have a problem reading in a large data-file
> if i just read it
> line by line it takes nearly ten minutes on a sparc twenty. my read
> block is the following
> <CODE>
> WHILE NOT eof(u1) DO BEGIN
> readf,u1,t_temp,foo,foo,foo,b_temp
> t_u = [t_u,t_temp]
> b_u = [b_u,b_temp]
> END
>
> Any ideas how to speed this up?
As long as you are doing this with a loop, this will be slow.
In general IDL is pretty slow with loops.
Option 1:
Read it in once the slow way and then save it as a binary file for
future reading, either using 'save' or 'assoc'.
Option2:
If you have many different text files you have to work with, you
might want to use a spawned or "call_external" called c or fortran
program to read them and then rewrite them as binaries.
Option3:
Another possibilty that will MIGHT be faster is to define a
structure that matches each row of data and then create an array
of this structure with one element for each line.
a=replicate{temp, t:double(0.0), foo1:double(0.0), foo2:double(0.0), $
foo3:double(0.0), bx:double(0.0), by:double(0.0), bz:double(0.0), $
nmuberofrows}
openr,u1,'text.file'
readf,u1,a
close,u1
t_u = a(*).t
b_xu = a(*).bx
b_yu = a(*).by
b_zu = a(*).bz
and then extract the info you really want out of the structure, "a"
You can find out the length of your files using some sort of system
command, like "wc" in UNIX. If you spawn this and write the output to
another file and read THAT to determine numberofrows you can make the
whole thing an automated program. Otherwise you will get an eof error if
numberofrows is too large, although the entire file will still be read
into "a".
option 2 is probably the quickest but also the most pain to implement.
It is worthwhile if this is a program where speed is critical and you
will have to read many different files every time you run.
----------
David Theil
home work
University of Colorado
3360 34th St. D campus box 389
Boulder, Colorado Boulder, Colorado
80301 80309
(303)449-3113 (303)492-0895
No guts, no glory.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
