Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
Re: Data management... but Execution halts!!! T.T
I'd guess that the number of data elements isn't quite what you think it is. Why don't you show us the output from the help command and the error message? regards, Greg
Sorry, I forgot to mention it. The data showed below is only small part of it. This data file contains 28 columns: location, year, month, and day, + each hour. There are 72 locations, and a location contains data with 2192 rows. Therefore 2192 * 72 = 157824 rows total. The error message was: DATA INT = Array[28] REFORM: New subscripts must not change the number elements in DATA. Execution halted at: TEMP1 27 D:\MODIS_Documents\MetData\IDLPractice\temp1.pro It seems that IDL reads only first rows. What do you think? Do you have any suggestions? Thanks. Bapts wrote: > kim20026@gmail.com a écrit : > >> I requested some meteological data to a governmental institute. To >> reduce the file size, they sent me this way... >> >> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17| >> ... >> >> This is only one line, but contains a matrix of 28 columns * 157824L >> rows. As you guys may know, Excel can only 65,000 rows approximately, >> and I need to divide this into three to read this file correctly. >> >> I have tried like this so far, but everytime I try, I have the same >> error message and the program stops. >> >> >> ------------------------------------------ >> Pro temp1 >> >> ; Main working directory (location of IDL procedures and functions) >> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice' >> >> s='' >> Openr, 1, 'temp1.txt' >> Readf, 1, s >> print, strlen(s) >> close, 1 >> >> data=fix(strsplit(s, '|', /extract)) >> help, data >> data=reform(data, 28, 157824L) ; <- Execution halts right here >> everytime!!! >> print, data[*, 0] >> print, data[*, 1] >> >> for i=0L, 157824-1 do begin >> if i mod 52608L eq 0 then begin >> close, /all >> file_no=i / 52608L + 1 ; 1, 2, 3 >> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt' >> ;ex) out1.txt >> endif >> printf, 1, strjoin(string(data[*, i], format='(I4)'), '') >> >> endfor >> close, /all >> end >> ------------------------------------------------------------ ------------ >> >> Please let me know if you have any solutions (or suggestions at least. >> It's urgent!! T.T) > > the data you've posted doesn't show 28 columns as you say in the text, > but 224 number separated with |'s. There is no reason for IDL not to > stop with an error when you try to reform an 224 element array into a > 28x157842 one...
Judging by the fragment you show, I'm a bit surprised that it stopped after exactly 28 columns, since I can't see anything different there. But what happens if you try swapping the line s='' for this: s=strarr(10) and then put in a break and check the first few elements - does each contain your 28 columns? If so, then try s=strarr(157824). You'll have to change the strsplit part (I don't think it can handle arrays), but it can be done with strpos and strmid. Once you have this wonderful data array, you could try plotting it - you might be surprised that IDL can do more than prepare data for Excel... regards, Greg
Do people not use READ_ASCII()? Is it really slow, or tempermental or something? I saved the original poster's data to a text file, and this seemed to work for me: data = read_ascii('file.txt', delimiter='|') There's a NaN on the end that I'd have to chop off, but that's easily done, but it also makes me wonder if the function doesn't work in a lot of cases maybe? I confess i've never used it myself before now, but that's b/c i never knew it existed until a few minutes ago :) Jeff kim20026@gmail.com wrote: > I requested some meteological data to a governmental institute. To > reduce the file size, they sent me this way... > > 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17| > ... > > This is only one line, but contains a matrix of 28 columns * 157824L > rows. As you guys may know, Excel can only 65,000 rows approximately, > and I need to divide this into three to read this file correctly. > > I have tried like this so far, but everytime I try, I have the same > error message and the program stops. > > > ------------------------------------------ > Pro temp1 > > ; Main working directory (location of IDL procedures and functions) > WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice' > > s='' > Openr, 1, 'temp1.txt' > Readf, 1, s > print, strlen(s) > close, 1 > > data=fix(strsplit(s, '|', /extract)) > help, data > data=reform(data, 28, 157824L) ; <- Execution halts right here > everytime!!! > print, data[*, 0] > print, data[*, 1] > > for i=0L, 157824-1 do begin > if i mod 52608L eq 0 then begin > close, /all > file_no=i / 52608L + 1 ; 1, 2, 3 > openw, 1, 'out'+string(file_no, format='(I1)')+'.txt' > ;ex) out1.txt > endif > printf, 1, strjoin(string(data[*, i], format='(I4)'), '') > > endfor > close, /all > end > ------------------------------------------------------------ ------------ > > Please let me know if you have any solutions (or suggestions at least. > It's urgent!! T.T)
Jeff N. wrote: > Do people not use READ_ASCII()? Is it really slow, or tempermental or > something? I saved the original poster's data to a text file, and this > seemed to work for me: I'd never noticed or used read_ascii either. Thanks for pointing it out. I would have read the data in question like this: line = '' openr, lun, 'file.dat', /get_lun readf, lun, line free_lun, lun data = float(strsplit(line, '|', /extract)) Mike
I wrote: > I would have read the data in question like this: Which is just what the original poster did, so nothing original...
I would have too. In fact that's what sent me off looking for a function like read_ascii() in the first place. I've done that so much in the past I started thinking about just putting code like that into my own function to generically read data from a text file, then it occurred to me that maybe the folks at ITTVIS had thought of the same idea :) Mike wrote: > I wrote: >> I would have read the data in question like this: > > Which is just what the original poster did, so nothing original...
The problem with read_ascii is the template. If your file has anything other than a trivial format (as this one does) - i.e. a header of some sort, or mixed types, you need to use a template. And you can't (reasonably) generate this programmatically, but only via the ascii_template GUI function. Try it, then you'll want to learn the other way... regards, Greg Jeff N. wrote: > Do people not use READ_ASCII()? Is it really slow, or tempermental or > something? I saved the original poster's data to a text file, and this > seemed to work for me: > > data = read_ascii('file.txt', delimiter='|') > > There's a NaN on the end that I'd have to chop off, but that's easily > done, but it also makes me wonder if the function doesn't work in a lot > of cases maybe? I confess i've never used it myself before now, but > that's b/c i never knew it existed until a few minutes ago :) > > Jeff > > > > kim20026@gmail.com wrote: >> I requested some meteological data to a governmental institute. To >> reduce the file size, they sent me this way... >> >> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17| >> ... >> >> This is only one line, but contains a matrix of 28 columns * 157824L >> rows. As you guys may know, Excel can only 65,000 rows approximately, >> and I need to divide this into three to read this file correctly. >> >> I have tried like this so far, but everytime I try, I have the same >> error message and the program stops. >> >> >> ------------------------------------------ >> Pro temp1 >> >> ; Main working directory (location of IDL procedures and functions) >> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice' >> >> s='' >> Openr, 1, 'temp1.txt' >> Readf, 1, s >> print, strlen(s) >> close, 1 >> >> data=fix(strsplit(s, '|', /extract)) >> help, data >> data=reform(data, 28, 157824L) ; <- Execution halts right here >> everytime!!! >> print, data[*, 0] >> print, data[*, 1] >> >> for i=0L, 157824-1 do begin >> if i mod 52608L eq 0 then begin >> close, /all >> file_no=i / 52608L + 1 ; 1, 2, 3 >> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt' >> ;ex) out1.txt >> endif >> printf, 1, strjoin(string(data[*, i], format='(I4)'), '') >> >> endfor >> close, /all >> end >> ------------------------------------------------------------ ------------ >> >> Please let me know if you have any solutions (or suggestions at least. >> It's urgent!! T.T)
And that last time I checked (a few years ago) the structure tags returned by read_ascii differed depending on if you had < 10 or >= 10 columns of data. This makes it less than ideal of your files have "optional" columns of data that occasionally bring the column count to 10 or more. < 10 -> data.field1 data.field2 ... >=10 -> data.field01 data.field02 ... -Rick greg michael wrote: > The problem with read_ascii is the template. If your file has anything > other than a trivial format (as this one does) - i.e. a header of some > sort, or mixed types, you need to use a template. And you can't > (reasonably) generate this programmatically, but only via the > ascii_template GUI function. Try it, then you'll want to learn the > other way... > > regards, > Greg > > > Jeff N. wrote: >> Do people not use READ_ASCII()? Is it really slow, or tempermental or >> something? I saved the original poster's data to a text file, and this >> seemed to work for me: >> >> data = read_ascii('file.txt', delimiter='|') >> >> There's a NaN on the end that I'd have to chop off, but that's easily >> done, but it also makes me wonder if the function doesn't work in a lot >> of cases maybe? I confess i've never used it myself before now, but >> that's b/c i never knew it existed until a few minutes ago :) >> >> Jeff >> >> >> >> kim20026@gmail.com wrote: >>> I requested some meteological data to a governmental institute. To >>> reduce the file size, they sent me this way... >>> >>> 90|2000|1|1|25|25|26|26|26|25|26|28|31|34|36|39|40|38|38|37| 35|33|35|37|34|34|35|29|90|2000|1|2|30|32|29|28|31|33|55|65| 74|68|72|75|71|83|85|77|69|60|51|43|41|34|34|29|90|2000|1|3| 25|20|12|9|8|5|-2|-3|14|27|31|42|51|54|48|51|37|28|15|11|6|- 12|0|-13|90|2000|1|4|-7|-5|-8|-10|-21|-21|-13|-18|-2|23|36|4 0|50|49|42|37|29|22|9|3|0|0|5|4|90|2000|1|5|1|4|12|10|15|15| 9|6|0|-1|1|3|5|4|4|6|8|8|10|9|9|13|13|17|90|2000|1|6|17|18|2 3|23|24|28|28|27|25|26|45|73|87|90|81|72|58|54|39|19|-2|-11| -17|-26|90|2000|1|7|-31|-39|-43|-55|-67|-77|-85|-86|-78|-71| -63|-63|-52|-43|-39|-41|-49|-51|-53|-54|-58|-53|-54|-57|90|2 000|1|8|-57|-52|-49|-48|-45|-39|-36|-30|-25|-18|-11|-8|0|2|1 2|7|-3|-7|-7|-7|-6|-16|-15|-17| >>> ... >>> >>> This is only one line, but contains a matrix of 28 columns * 157824L >>> rows. As you guys may know, Excel can only 65,000 rows approximately, >>> and I need to divide this into three to read this file correctly. >>> >>> I have tried like this so far, but everytime I try, I have the same >>> error message and the program stops. >>> >>> >>> ------------------------------------------ >>> Pro temp1 >>> >>> ; Main working directory (location of IDL procedures and functions) >>> WorkDir = 'D:\MODIS_Documents\MetData\IDLPractice' >>> >>> s='' >>> Openr, 1, 'temp1.txt' >>> Readf, 1, s >>> print, strlen(s) >>> close, 1 >>> >>> data=fix(strsplit(s, '|', /extract)) >>> help, data >>> data=reform(data, 28, 157824L) ; <- Execution halts right here >>> everytime!!! >>> print, data[*, 0] >>> print, data[*, 1] >>> >>> for i=0L, 157824-1 do begin >>> if i mod 52608L eq 0 then begin >>> close, /all >>> file_no=i / 52608L + 1 ; 1, 2, 3 >>> openw, 1, 'out'+string(file_no, format='(I1)')+'.txt' >>> ;ex) out1.txt >>> endif >>> printf, 1, strjoin(string(data[*, i], format='(I4)'), '') >>> >>> endfor >>> close, /all >>> end >>> ------------------------------------------------------------ ------------ >>> >>> Please let me know if you have any solutions (or suggestions at least. >>> It's urgent!! T.T) >
greg michael wrote: > The problem with read_ascii is the template. If your file has anything > other than a trivial format (as this one does) - i.e. a header of some > sort, or mixed types, you need to use a template. And you can't > (reasonably) generate this programmatically, but only via the > ascii_template GUI function. Try it, then you'll want to learn the > other way... Nah, use it once, use 'help, /structure, template' to figure out what the returned struct looks like, and then put that struct in your code. Works like a charm. There are some formats that require more coding, but read_ascii() with ascii_template() _once_ isn't too bad. Maarten
Wrote a reply already... seems not to have turned up :( Yes, actually that seems to work fine - it's especially useful that it can handle mixed-type tables. I take back (some of) the bad things I said about read_ascii! You can use this to generate the code for copy/paste: pro what_ascii_template t=ascii_template() tags=tag_names(t) n=n_tags(t) print,'template={$' for i=0,n-1 do begin if size(t.(i),/type) eq 7 then s="'"+t.(i)+"'" else s=strtrim(t.(i),2) s=strjoin(s,',') if n_elements(t.(i)) gt 1 then s='['+s+']' print,' ',tags[i],'=',s,(i lt n-1)?',$':'}' endfor end IDL> what_ascii_template template={$ VERSION=1.00000,$ DATASTART=0,$ DELIMITER=,$ MISSINGVALUE=NaN,$ COMMENTSYMBOL='#',$ FIELDCOUNT=7,$ FIELDTYPES=[4,3,4,4,3,4,4],$ FIELDNAMES=['FIELD1','FIELD2','FIELD3','FIELD4','FIELD5','FI ELD6','FIELD7'],$ FIELDLOCATIONS=[0,18,22,34,52,58,70],$ FIELDGROUPS=[0,1,2,3,4,5,6]} Think I'm converted - thanks! Solves Rick's problem, too, since you can modify or generate the field names yourself. Greg Maarten wrote: > greg michael wrote: >> The problem with read_ascii is the template. If your file has anything >> other than a trivial format (as this one does) - i.e. a header of some >> sort, or mixed types, you need to use a template. And you can't >> (reasonably) generate this programmatically, but only via the >> ascii_template GUI function. Try it, then you'll want to learn the >> other way... > > Nah, use it once, use 'help, /structure, template' to figure out what > the > returned struct looks like, and then put that struct in your code. > Works > like a charm. > > There are some formats that require more coding, but read_ascii() with > ascii_template() _once_ isn't too bad. > > Maarten