| Input line too long??? [message #20682] |
Wed, 19 July 2000 00:00  |
David Bowman
Messages: 10
Registered: July 2000
|
Junior Member |
|
|
Greetings all,
I have a data file that essentially contains a very long vector of
floating point numbers. The file (written by a trusty old FORTRAN
program) gives the number of elements in the record on the first line.
The second line contains all the data, with each element separated by
two spaces. For example:
.123348 .126664 .130130 .133735 .137486 and so on.
Now in the past I've read this data with no problem simply using
x=fltarr(n)
readf,1,x,format='(nF0)'
where n is the number of elements in the vector. However, I've
recently tried to read in a much larger vector, and IDL chokes, saying:
% Input line is too long for input buffer of 32767 characters.
The datafile in question contains 5856 elements (five of which are in
my little example above). All told, there are 55278 characters in the
line in question.
Nothing I've tried works... Reading in elements by element, with
formatting, without formatting... I've even tried using the BUFSIZE
keyword to OPEN, with no result. The IDL help says that I'm limited to
reading in files smaller than 2^31 bytes, but I'm certainly well below
that.
I'd love to simply rewrite the I/O section of the program that made the
file to begin with, but I didn't write it and I don't have the source
code.
Any Ideas? I'm using version 5.3 on a Mac G3 with 128 Mb RAM.
Thanks in advance...
David
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-=-=-=-=-=-=-=
David Bowman "Smash forehead on keyboard to continue."
Laboratoire Tectonique
Institut de Physique du Globe - Paris
4 Place Jussieu
75252 Paris Cedex 05
France
|
|
|
|
| Re: Input line too long??? [message #20753 is a reply to message #20682] |
Thu, 20 July 2000 00:00  |
promashkin
Messages: 169
Registered: December 1999
|
Senior Member |
|
|
David Bowman wrote:
>
> [[ This message was both posted and mailed: see
> the "To," "Cc," and "Newsgroups" headers for details. ]]
>
> Pavel,
> Thanks, that worked like a charm! Is there any way of doing the
> same operation when the records are not all of the same length? Just
> curious.
David,
I tried to play with different byte lengths per record and it seems
pretty simple, as long as you know when the record length changes and
those changes are repetitive. Then, you could read the long string in
increments, like this:
;if a string is like:
;.1111 .2222 .3333 .4444 .55 .66 .1111 .2222 .3333 .4444 .55 .66 ... etc,
;then
x = strarr(4)
x[*] = '123456' ; 6-byte field, 4 of them
y = ['1234', '1234'] ; 4-byte field, 2 of them
while not EOF(unit) do begin
readu, unit, x
readu, unit, y
; Reassign X and Y to permanent array(s) to take them out of the loop
endwhile
If varying field lengths were not regularly distributed in the string,
I'd probably just open that file in a word processor and save it as text
with line breaks, then read formatted (READF) as floating point values.
Good solution for a lazy IDLer :-)
Cheers,
Pavel
|
|
|
|
| Re: Input line too long??? [message #20760 is a reply to message #20682] |
Thu, 20 July 2000 00:00 |
David Bowman
Messages: 10
Registered: July 2000
|
Junior Member |
|
|
[[ This message was both posted and mailed: see
the "To," "Cc," and "Newsgroups" headers for details. ]]
Pavel,
Thanks, that worked like a charm! Is there any way of doing the
same operation when the records are not all of the same length? Just
curious.
david
In article <3975D9F7.7B28AC1B@cmdl.noaa.gov>, Pavel Romashkin
<promashkin@cmdl.noaa.gov> wrote:
> Hi David,
> I played with your example and I think I have a solution. The input
> buffer limitation is only in effect if you read formatted. I made a file
> by repeating your example line 2000 times, obtaining 10000 element line.
> Each record in your line is 8 bytes long, including a space. Try reading
> unformatted and it works perfectly:
>
> ; Make fake data file with 1 line
> ; with 10000 data entries and open it.
> openr, unit, 'test.txt', /get_lun
> ; Since you know the number of data points, make X:
> x = strarr(10000)
> ; Fill X with 8 byte strings.
> x[*] = '12345678'
> readu, unit, x
>
> Now you can convert X into float and use it.
> Cheers,
> Pavel
>
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-=-=-=-=-=-=-=
David Bowman "Smash forehead on keyboard to continue."
Laboratoire Tectonique
Institut de Physique du Globe - Paris
4 Place Jussieu
75252 Paris Cedex 05
France
|
|
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
