| Re: Read large ascii file quickly [message #74293 is a reply to message #74291] |
Thu, 06 January 2011 16:06   |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Paulo Penteado writes:
> It is what I would do for a large file, except that I do not see why
> make data 1D then reform it, instead of making it 2D from the start.
I don't know. I'm writing a book. Pedantry, I guess. :-)
> Since the file is large and he only wants one column out of 6, I guess
> the above would be better than the one-liner
>
> data=reform((read_csv('file.dat')).(5),5000,5000)
>
> Which would use more memory, and I expect would be slower.
I would expect it to be about as slow as the loop, but
I've never tried it.
> Anyway, if the same file is going to be read several times, it may be
> useful to convert it to some binary format, like Ken suggested (might
> even be a savefile).
Yeah, people who put big data sets in columns should
be shot, but sometimes you just have to play the hand
you are dealt. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
