| Re: Read large ascii file quickly [message #74275 is a reply to message #74265] |
Fri, 07 January 2011 15:19   |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Kenneth P. Bowman writes:
> But reading ASCII will always be noticeably slower than
> reading the equivalent binary because of the overhead involved
> in converting from ASCII to the internal binary representation.
> Binary formats, on the other hand, basically just move bytes
> from the file to memory.
No argument from me here. But "slower" doesn't mean
minutes slower for well-formatted ASCII files. Possibly
some number of seconds slower. But, as someone pointed
out, I don't have a lot of experience with extremely
large ASCII files, as people with big data sets generally
use binary formats.
> ASCII files are quite portable, but they are usually inefficient
> in terms of storage size. They also rarely carry adequate metadata
> within the file itself, but that is a failing of the file's
> author rather than the format, per se.
Well, binary files force good documentation practices because
without it you are utterly and totally without hope. :-)
> Ah, remember the good old days when you had to try to read
> a binary file written on a 60-bit CDC machine on a byte-
> oriented machine. :-)
Before my time, Ken. ;-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
