| Re: Read large ascii file quickly [message #74294 is a reply to message #74293] |
Thu, 06 January 2011 15:29   |
penteado
Messages: 866
Registered: February 2018
|
Senior Member
Administrator |
|
|
On Jan 6, 9:06 pm, David Fanning <n...@dfanning.com> wrote:
> Paul Magdon writes:
>> I print the counter just to make sure something is working.
>
>> Here are a couple of lines of the ascii:
>
>> 0.050249,0.058117,0.029660,0.097091,0.358593,1
>> 0.051483,0.061990,0.030229,0.095512,0.371539,1
>> 0.055431,0.062700,0.031565,0.090342,0.382935,1
>
> Well, then I would do it like this:
>
> OpenR, lun, 'file.dat', /Get_Lun
> data = Intarr(5000*5000)
> ReadF, lun, data, Format='(44x,I0)'
> Free_Lun, lun
> data = Reform(data, 5000, 5000)
It is what I would do for a large file, except that I do not see why
make data 1D then reform it, instead of making it 2D from the start.
Since the file is large and he only wants one column out of 6, I guess
the above would be better than the one-liner
data=reform((read_csv('file.dat')).(5),5000,5000)
Which would use more memory, and I expect would be slower.
Anyway, if the same file is going to be read several times, it may be
useful to convert it to some binary format, like Ken suggested (might
even be a savefile).
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
