| Re: Reading large Ascii files [message #27631] |
Tue, 30 October 2001 09:24 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
"Martin Downing" <martin.downing@ntlworld.com> writes:
> Isa,
> I cant reproduce your problem,
> here is the output I get from the code below which does outpur/input of
> 12500 floats in one line
> print, !version
>
> { x86 Win32 Windows Microsoft Windows 5.5 Beta Jun 20 2001 32 64}
>
> IDL> ascii_io_large_test
> values read incorrectly = 0
> line length read = 100000
Hi Isa--
Your problem is definitely the lack of ability of your IDL version to
read strings longer than 32767 characters. If you are stuck with your
existing version of IDL, then you will have to do something else, like
READU to read the data in, and then parse it in memory.
The reason Martin's example worked was that he was using IDL 5.5.
According to the release notes for IDL 5.5, the 32767 character limit
for strings was lifted, so you wouldn't have that particular problem
if you upgraded. Now if you have a Mac and need long strings then you
might be in trouble...
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Reading large Ascii files [message #27632 is a reply to message #27631] |
Tue, 30 October 2001 08:48  |
Isa Usman
Messages: 13
Registered: October 2001
|
Junior Member |
|
|
Martin,
I decided to run your code on my machine with the only change being the
file path and it is giving me a buffer error. This means that probably the
problem is machine specific. See below
Thanks
Isa
{ x86 Win32 Windows 5.3.1 Feb 23 2000}
IDL> .COMPILE "D:\Idl readfile\Programs\ascii_io_large_test.pro"
% Compiled module: ASCII_IO_LARGE_TEST.
IDL> ascii_io_large_test
% Format fills output buffer. Re-write as multiple I/O statements.
% Execution halted at: ASCII_IO_LARGE_TEST 11 D:\Idl
readfile\Programs\ascii_io_large_test.pro
% $MAIN$
----------------------------------------------------------
"Martin Downing" <martin.downing@ntlworld.com> wrote in message
news:ojzD7.29678$GA.3999822@news2-win.server.ntlworld.com...
> Isa,
> I cant reproduce your problem,
> here is the output I get from the code below which does outpur/input of
> 12500 floats in one line
> print, !version
>
> { x86 Win32 Windows Microsoft Windows 5.5 Beta Jun 20 2001 32 64}
>
> IDL> ascii_io_large_test
> values read incorrectly = 0
> line length read = 100000
>
>
> regards
>
> Martin
>
> =================================
> pro ascii_io_large_test
>
> fn = !fpaths.tmp + "aio_test.txt"
>
> ; create output
> a = findgen(12500)/1000
>
> ; write test file
> openw, lun, fn, /get_lun, width = 500000L
> ; write two lines worth
> printf, lun, a,format='(12500F8.3)'
> printf, lun, a, format='(12500F8.3)'
> close, lun
>
> ; define input variables
> b= fltarr(12500)
> line = ""
>
> ; read test file
> openr, lun, fn, /get_lun;, width = 10000L
> readf,lun,format='(12500F8.3)',b
> readf,lun,line
> close , lun
>
> ; analyse data
> ids = where(a ne b, count)
> print, "values read incorrectly =", count
> print,"line length read =", strlen(line)
>
> end
> =================================
> --
> ----------------------------------------
> Martin Downing,
> Clinical Research Physicist,
> Grampian Orthopaedic RSA Research Centre,
> Woodend Hospital, Aberdeen, AB15 6LS.
>
> "Isa Usman" <I.Usman@rl.ac.uk> wrote in message
> news:9rma6a$gbk@newton.cc.rl.ac.uk...
>> Hello,
>>
>> I am trying to read a single row from a large file. The amout of numbers
> per
>> row is about 12500. So in IDL i have the following lines;
>>
>> readf,lun,format='(500F8.3)',ch1_val_s
>>
>> so that i only read 500 values at a time and it gives me the error;
>>
>> % Input line is too long for input buffer of 32767 characters.
>>
>> From what i can gather, IDL is trying to read the whole line first
before
> it
>> puts it into the array. Can anybody suggest a way round this please.
>>
>> Thanks
>>
>> Isa Usman
>>
>>
>>
>
>
>
>
|
|
|
|
| Re: Reading large Ascii files [message #27633 is a reply to message #27632] |
Tue, 30 October 2001 08:30 |
Med Bennett
Messages: 109
Registered: April 1997
|
Senior Member |
|
|
Isa Usman wrote:
> Hello,
>
> I am trying to read a single row from a large file. The amout of numbers per
> row is about 12500. So in IDL i have the following lines;
>
> readf,lun,format='(500F8.3)',ch1_val_s
>
> so that i only read 500 values at a time and it gives me the error;
>
> % Input line is too long for input buffer of 32767 characters.
>
> From what i can gather, IDL is trying to read the whole line first before it
> puts it into the array. Can anybody suggest a way round this please.
>
> Thanks
>
> Isa Usman
Yes - I think in this situation you will have to use readu to read the whole
file in to a very long byte array and then use where and the various string
commands to extract the data.
|
|
|
|
| Re: Reading large Ascii files [message #27634 is a reply to message #27633] |
Tue, 30 October 2001 08:28 |
Pavel A. Romashkin
Messages: 531
Registered: November 2000
|
Senior Member |
|
|
How is ch1_val_s defined? Is it defined as a string?
There have been at least two times when READF buffer overflows were
brought up. Search Google newsgroup for "input buffer of 32767". In
brief, one solution is to use READU.
Cheers,
Pavel
Isa Usman wrote:
>
> Hello,
>
> I am trying to read a single row from a large file. The amout of numbers per
> row is about 12500. So in IDL i have the following lines;
>
> readf,lun,format='(500F8.3)',ch1_val_s
>
> so that i only read 500 values at a time and it gives me the error;
>
> % Input line is too long for input buffer of 32767 characters.
>
> From what i can gather, IDL is trying to read the whole line first before it
> puts it into the array. Can anybody suggest a way round this please.
>
> Thanks
>
> Isa Usman
|
|
|
|
| Re: Reading large Ascii files [message #27638 is a reply to message #27634] |
Tue, 30 October 2001 07:01 |
Martin Downing
Messages: 136
Registered: September 1998
|
Senior Member |
|
|
Isa,
I cant reproduce your problem,
here is the output I get from the code below which does outpur/input of
12500 floats in one line
print, !version
{ x86 Win32 Windows Microsoft Windows 5.5 Beta Jun 20 2001 32 64}
IDL> ascii_io_large_test
values read incorrectly = 0
line length read = 100000
regards
Martin
=================================
pro ascii_io_large_test
fn = !fpaths.tmp + "aio_test.txt"
; create output
a = findgen(12500)/1000
; write test file
openw, lun, fn, /get_lun, width = 500000L
; write two lines worth
printf, lun, a,format='(12500F8.3)'
printf, lun, a, format='(12500F8.3)'
close, lun
; define input variables
b= fltarr(12500)
line = ""
; read test file
openr, lun, fn, /get_lun;, width = 10000L
readf,lun,format='(12500F8.3)',b
readf,lun,line
close , lun
; analyse data
ids = where(a ne b, count)
print, "values read incorrectly =", count
print,"line length read =", strlen(line)
end
=================================
--
----------------------------------------
Martin Downing,
Clinical Research Physicist,
Grampian Orthopaedic RSA Research Centre,
Woodend Hospital, Aberdeen, AB15 6LS.
"Isa Usman" <I.Usman@rl.ac.uk> wrote in message
news:9rma6a$gbk@newton.cc.rl.ac.uk...
> Hello,
>
> I am trying to read a single row from a large file. The amout of numbers
per
> row is about 12500. So in IDL i have the following lines;
>
> readf,lun,format='(500F8.3)',ch1_val_s
>
> so that i only read 500 values at a time and it gives me the error;
>
> % Input line is too long for input buffer of 32767 characters.
>
> From what i can gather, IDL is trying to read the whole line first before
it
> puts it into the array. Can anybody suggest a way round this please.
>
> Thanks
>
> Isa Usman
>
>
>
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
