| Re: Data management... but Execution halts!!! T.T [message #51292] |
Wed, 15 November 2006 07:50  |
![Maarten[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Maarten[1] is currently offline")
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
greg michael wrote:
> The problem with read_ascii is the template. If your file has anything
> other than a trivial format (as this one does) - i.e. a header of some
> sort, or mixed types, you need to use a template. And you can't
> (reasonably) generate this programmatically, but only via the
> ascii_template GUI function. Try it, then you'll want to learn the
> other way...
Nah, use it once, use 'help, /structure, template' to figure out what
the
returned struct looks like, and then put that struct in your code.
Works
like a charm.
There are some formats that require more coding, but read_ascii() with
ascii_template() _once_ isn't too bad.
Maarten
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51309 is a reply to message #51292] |
Tue, 14 November 2006 11:28  |
Rick Towler
Messages: 821
Registered: August 1998
|
Senior Member |
|
|
And that last time I checked (a few years ago) the structure tags
returned by read_ascii differed depending on if you had < 10 or >= 10
columns of data. This makes it less than ideal of your files have
"optional" columns of data that occasionally bring the column count to
10 or more.
< 10 -> data.field1 data.field2 ...
>=10 -> data.field01 data.field02 ...
-Rick
greg michael wrote:
> The problem with read_ascii is the template. If your file has anything
> other than a trivial format (as this one does) - i.e. a header of some
> sort, or mixed types, you need to use a template. And you can't
> (reasonably) generate this programmatically, but only via the
> ascii_template GUI function. Try it, then you'll want to learn the
> other way...
>
> regards,
> Greg
>
>
> Jeff N. wrote:
>> Do people not use READ_ASCII()? Is it really slow, or tempermental or
>> something? I saved the original poster's data to a text file, and this
>> seemed to work for me:
>>
>> data = read_ascii('file.txt', delimiter='|')
>>
>> There's a NaN on the end that I'd have to chop off, but that's easily
>> done, but it also makes me wonder if the function doesn't work in a lot
>> of cases maybe? I confess i've never used it myself before now, but
>> that's b/c i never knew it existed until a few minutes ago :)
>>
>> Jeff
>>
>>
>>
>> kim20026@gmail.com wrote:
>>> I requested some meteological data to a governmental institute. To
>>> reduce the file size, they sent me this way...
>>>
>>> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17|
>>> ...
>>>
>>> This is only one line, but contains a matrix of 28 columns * 157824L
>>> rows. As you guys may know, Excel can only 65,000 rows approximately,
>>> and I need to divide this into three to read this file correctly.
>>>
>>> I have tried like this so far, but everytime I try, I have the same
>>> error message and the program stops.
>>>
>>>
>>> ------------------------------------------
>>> Pro temp1
>>>
>>> ; Main working directory (location of IDL procedures and functions)
>>> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice'
>>>
>>> s=''
>>> Openr, 1, 'temp1.txt'
>>> Readf, 1, s
>>> print, strlen(s)
>>> close, 1
>>>
>>> data=fix(strsplit(s, '|', /extract))
>>> help, data
>>> data=reform(data, 28, 157824L) ; <- Execution halts right here
>>> everytime!!!
>>> print, data[*, 0]
>>> print, data[*, 1]
>>>
>>> for i=0L, 157824-1 do begin
>>> if i mod 52608L eq 0 then begin
>>> close, /all
>>> file_no=i / 52608L + 1 ; 1, 2, 3
>>> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt'
>>> ;ex) out1.txt
>>> endif
>>> printf, 1, strjoin(string(data[*, i], format='(I4)'), '')
>>>
>>> endfor
>>> close, /all
>>> end
>>> ------------------------------------------------------------ ------------
>>>
>>> Please let me know if you have any solutions (or suggestions at least.
>>> It's urgent!! T.T)
>
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51317 is a reply to message #51309] |
Tue, 14 November 2006 07:29 |
greg michael
Messages: 163
Registered: January 2006
|
Senior Member |
|
|
The problem with read_ascii is the template. If your file has anything
other than a trivial format (as this one does) - i.e. a header of some
sort, or mixed types, you need to use a template. And you can't
(reasonably) generate this programmatically, but only via the
ascii_template GUI function. Try it, then you'll want to learn the
other way...
regards,
Greg
Jeff N. wrote:
> Do people not use READ_ASCII()? Is it really slow, or tempermental or
> something? I saved the original poster's data to a text file, and this
> seemed to work for me:
>
> data = read_ascii('file.txt', delimiter='|')
>
> There's a NaN on the end that I'd have to chop off, but that's easily
> done, but it also makes me wonder if the function doesn't work in a lot
> of cases maybe? I confess i've never used it myself before now, but
> that's b/c i never knew it existed until a few minutes ago :)
>
> Jeff
>
>
>
> kim20026@gmail.com wrote:
>> I requested some meteological data to a governmental institute. To
>> reduce the file size, they sent me this way...
>>
>> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17|
>> ...
>>
>> This is only one line, but contains a matrix of 28 columns * 157824L
>> rows. As you guys may know, Excel can only 65,000 rows approximately,
>> and I need to divide this into three to read this file correctly.
>>
>> I have tried like this so far, but everytime I try, I have the same
>> error message and the program stops.
>>
>>
>> ------------------------------------------
>> Pro temp1
>>
>> ; Main working directory (location of IDL procedures and functions)
>> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice'
>>
>> s=''
>> Openr, 1, 'temp1.txt'
>> Readf, 1, s
>> print, strlen(s)
>> close, 1
>>
>> data=fix(strsplit(s, '|', /extract))
>> help, data
>> data=reform(data, 28, 157824L) ; <- Execution halts right here
>> everytime!!!
>> print, data[*, 0]
>> print, data[*, 1]
>>
>> for i=0L, 157824-1 do begin
>> if i mod 52608L eq 0 then begin
>> close, /all
>> file_no=i / 52608L + 1 ; 1, 2, 3
>> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt'
>> ;ex) out1.txt
>> endif
>> printf, 1, strjoin(string(data[*, i], format='(I4)'), '')
>>
>> endfor
>> close, /all
>> end
>> ------------------------------------------------------------ ------------
>>
>> Please let me know if you have any solutions (or suggestions at least.
>> It's urgent!! T.T)
|
|
|
|
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51326 is a reply to message #51325] |
Mon, 13 November 2006 10:40 |
Mike[2]
Messages: 99
Registered: December 2005
|
Member |
|
|
Jeff N. wrote:
> Do people not use READ_ASCII()? Is it really slow, or tempermental or
> something? I saved the original poster's data to a text file, and this
> seemed to work for me:
I'd never noticed or used read_ascii either. Thanks for pointing it
out.
I would have read the data in question like this:
line = ''
openr, lun, 'file.dat', /get_lun
readf, lun, line
free_lun, lun
data = float(strsplit(line, '|', /extract))
Mike
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51333 is a reply to message #51326] |
Mon, 13 November 2006 07:43 |
Jeff N.
Messages: 120
Registered: April 2005
|
Senior Member |
|
|
Do people not use READ_ASCII()? Is it really slow, or tempermental or
something? I saved the original poster's data to a text file, and this
seemed to work for me:
data = read_ascii('file.txt', delimiter='|')
There's a NaN on the end that I'd have to chop off, but that's easily
done, but it also makes me wonder if the function doesn't work in a lot
of cases maybe? I confess i've never used it myself before now, but
that's b/c i never knew it existed until a few minutes ago :)
Jeff
kim20026@gmail.com wrote:
> I requested some meteological data to a governmental institute. To
> reduce the file size, they sent me this way...
>
> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17|
> ...
>
> This is only one line, but contains a matrix of 28 columns * 157824L
> rows. As you guys may know, Excel can only 65,000 rows approximately,
> and I need to divide this into three to read this file correctly.
>
> I have tried like this so far, but everytime I try, I have the same
> error message and the program stops.
>
>
> ------------------------------------------
> Pro temp1
>
> ; Main working directory (location of IDL procedures and functions)
> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice'
>
> s=''
> Openr, 1, 'temp1.txt'
> Readf, 1, s
> print, strlen(s)
> close, 1
>
> data=fix(strsplit(s, '|', /extract))
> help, data
> data=reform(data, 28, 157824L) ; <- Execution halts right here
> everytime!!!
> print, data[*, 0]
> print, data[*, 1]
>
> for i=0L, 157824-1 do begin
> if i mod 52608L eq 0 then begin
> close, /all
> file_no=i / 52608L + 1 ; 1, 2, 3
> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt'
> ;ex) out1.txt
> endif
> printf, 1, strjoin(string(data[*, i], format='(I4)'), '')
>
> endfor
> close, /all
> end
> ------------------------------------------------------------ ------------
>
> Please let me know if you have any solutions (or suggestions at least.
> It's urgent!! T.T)
|
|
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51337 is a reply to message #51336] |
Mon, 13 November 2006 01:10 |
kim20026
Messages: 54
Registered: November 2006
|
Member |
|
|
Sorry, I forgot to mention it. The data showed below is only small part
of it. This data file contains 28 columns: location, year, month, and
day, + each hour. There are 72 locations, and a location contains data
with 2192 rows. Therefore 2192 * 72 = 157824 rows total.
The error message was:
DATA INT = Array[28]
REFORM: New subscripts must not change the number elements in DATA.
Execution halted at: TEMP1 27
D:\MODIS_Documents\MetData\IDLPractice\temp1.pro
It seems that IDL reads only first rows.
What do you think? Do you have any suggestions? Thanks.
Bapts wrote:
> kim20026@gmail.com a écrit :
>
>> I requested some meteological data to a governmental institute. To
>> reduce the file size, they sent me this way...
>>
>> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17|
>> ...
>>
>> This is only one line, but contains a matrix of 28 columns * 157824L
>> rows. As you guys may know, Excel can only 65,000 rows approximately,
>> and I need to divide this into three to read this file correctly.
>>
>> I have tried like this so far, but everytime I try, I have the same
>> error message and the program stops.
>>
>>
>> ------------------------------------------
>> Pro temp1
>>
>> ; Main working directory (location of IDL procedures and functions)
>> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice'
>>
>> s=''
>> Openr, 1, 'temp1.txt'
>> Readf, 1, s
>> print, strlen(s)
>> close, 1
>>
>> data=fix(strsplit(s, '|', /extract))
>> help, data
>> data=reform(data, 28, 157824L) ; <- Execution halts right here
>> everytime!!!
>> print, data[*, 0]
>> print, data[*, 1]
>>
>> for i=0L, 157824-1 do begin
>> if i mod 52608L eq 0 then begin
>> close, /all
>> file_no=i / 52608L + 1 ; 1, 2, 3
>> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt'
>> ;ex) out1.txt
>> endif
>> printf, 1, strjoin(string(data[*, i], format='(I4)'), '')
>>
>> endfor
>> close, /all
>> end
>> ------------------------------------------------------------ ------------
>>
>> Please let me know if you have any solutions (or suggestions at least.
>> It's urgent!! T.T)
>
> the data you've posted doesn't show 28 columns as you say in the text,
> but 224 number separated with |'s. There is no reason for IDL not to
> stop with an error when you try to reform an 224 element array into a
> 28x157842 one...
|
|
|
|
|
|
|
|
| Re: Data management... but Execution halts!!! T.T [message #51388 is a reply to message #51292] |
Wed, 15 November 2006 10:22 |
greg michael
Messages: 163
Registered: January 2006
|
Senior Member |
|
|
Wrote a reply already... seems not to have turned up :(
Yes, actually that seems to work fine - it's especially useful that it
can handle mixed-type tables. I take back (some of) the bad things I
said about read_ascii! You can use this to generate the code for
copy/paste:
pro what_ascii_template
t=ascii_template()
tags=tag_names(t)
n=n_tags(t)
print,'template={$'
for i=0,n-1 do begin
if size(t.(i),/type) eq 7 then s="'"+t.(i)+"'" else
s=strtrim(t.(i),2)
s=strjoin(s,',')
if n_elements(t.(i)) gt 1 then s='['+s+']'
print,' ',tags[i],'=',s,(i lt n-1)?',$':'}'
endfor
end
IDL> what_ascii_template
template={$
VERSION=1.00000,$
DATASTART=0,$
DELIMITER=,$
MISSINGVALUE=NaN,$
COMMENTSYMBOL='#',$
FIELDCOUNT=7,$
FIELDTYPES=[4,3,4,4,3,4,4],$
FIELDNAMES=['FIELD1','FIELD2','FIELD3','FIELD4','FIELD5','FI ELD6','FIELD7'],$
FIELDLOCATIONS=[0,18,22,34,52,58,70],$
FIELDGROUPS=[0,1,2,3,4,5,6]}
Think I'm converted - thanks! Solves Rick's problem, too, since you can
modify or generate the field names yourself.
Greg
Maarten wrote:
> greg michael wrote:
>> The problem with read_ascii is the template. If your file has anything
>> other than a trivial format (as this one does) - i.e. a header of some
>> sort, or mixed types, you need to use a template. And you can't
>> (reasonably) generate this programmatically, but only via the
>> ascii_template GUI function. Try it, then you'll want to learn the
>> other way...
>
> Nah, use it once, use 'help, /structure, template' to figure out what
> the
> returned struct looks like, and then put that struct in your code.
> Works
> like a charm.
>
> There are some formats that require more coding, but read_ascii() with
> ascii_template() _once_ isn't too bad.
>
> Maarten
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
