| Read several files into one big data array? [message #89507] |
Wed, 22 October 2014 13:57  |
lucesmm
Messages: 26
Registered: October 2014
|
Junior Member |
|
|
I have tons of data files in the same folder. They are named correctly with the last 5 digits representing the order
Example
FILE_00001.dat
FILE_00002.dat
....
Within this files I have data formatted like this, The length of the data varies from file to file
_____________________________________
seconds Time data1 data2
189 00:03:09 111.20 770.62
3785 01:03:05 200.15 255.66
7345 02:02:25 198.83 779.16
10983 03:03:03 200.01 555.55
#Comments
#Comments
#Comments
____________________________________
I want to read all the files and storage the data in the following format
_________________________________________
data=[
00001 4189 00:03:09 111.20 770.62
00001 3785 01:03:05 200.15 255.66
00001 745 02:02:25 198.83 779.16
00001 10983 03:03:03 200.01 555.55
00002 532 02:56:00 888.01 998.20
00002 200 23:59:52 222.00 000.10
....]
_________________________________________
So this is what I have so far. But I am stock with getting the data from the title
PRO
files = file_search('\folder','*.dat', COUNT=nfiles)
for i=0, nfiles-1 do begin
nlines = FILE_LINES(files[i])
data = FLTARR(nlines)
OPENR, lunit, files[i], /get_lun
READF, lunit, data, FORMAT='(6(I, C(CHI.2, ':', CMI.2, ':', CSI.2),2F))'
CLOSE, lunit
FREE_LUN, lunit
endfor
|
|
|
|
| Re: Read several files into one big data array? [message #89508 is a reply to message #89507] |
Wed, 22 October 2014 14:23  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
lucesmm@gmail.com writes:
>
> I have tons of data files in the same folder. They are named correctly with the last 5 digits representing the order
> Example
> FILE_00001.dat
> FILE_00002.dat
> ....
> Within this files I have data formatted like this, The length of the data varies from file to file
> _____________________________________
> seconds Time data1 data2
> 189 00:03:09 111.20 770.62
> 3785 01:03:05 200.15 255.66
> 7345 02:02:25 198.83 779.16
> 10983 03:03:03 200.01 555.55
>
> #Comments
> #Comments
> #Comments
>
> ____________________________________
>
>
> I want to read all the files and storage the data in the following format
> _________________________________________
> data=[
> 00001 4189 00:03:09 111.20 770.62
> 00001 3785 01:03:05 200.15 255.66
> 00001 745 02:02:25 198.83 779.16
> 00001 10983 03:03:03 200.01 555.55
> 00002 532 02:56:00 888.01 998.20
> 00002 200 23:59:52 222.00 000.10
> ....]
> _________________________________________
> So this is what I have so far. But I am stock with getting the data from the title
> PRO
> files = file_search('\folder','*.dat', COUNT=nfiles)
>
> for i=0, nfiles-1 do begin
>
> nlines = FILE_LINES(files[i])
> data = FLTARR(nlines)
> OPENR, lunit, files[i], /get_lun
> READF, lunit, data, FORMAT='(6(I, C(CHI.2, ':', CMI.2, ':', CSI.2),2F))'
> CLOSE, lunit
> FREE_LUN, lunit
> endfor
I would add these two lines between your OPENR and READF lines:
header = ""
READF, lunit, header
If you want the name of each column, you could do this after you read
the header line:
names = StrSplit(header, /Extract)
The variable "names" will be a four-element array containing the names
of the four columns of data.
By the way, you don't need the "CLOSE, lunit" statement in your code.
FREE_LUN will close the file unit for you while it is freeing it up to
be used again.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| Re: Read several files into one big data array? [message #89510 is a reply to message #89509] |
Wed, 22 October 2014 14:55 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
lucesmm@gmail.com writes:
> I really don't care about the names of the columns, they are fixed for all the files. But when I referred to title, I meant the last 5 digits of the file's name, to be in a new column (first column). (i.e. FILE_00001.data -> extract the "00001" and storage it with the data in the first column.
Oh. Something like this, then:
locate_underscore = StrPos(filename, "_")
fileNumber = StrMid(filename, locate_underscore, locate_underscore+5)
> Also, my code is still not working, so I am curious if this will group the data from FILE_00001.dat with FILE_00002.data and so on?
I doubt it, but can't tell without some code that tries to do it. :-)
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| Re: Read several files into one big data array? [message #89512 is a reply to message #89511] |
Wed, 22 October 2014 15:49 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
lucesmm@gmail.com writes:
> The title works great!. But now I having ploblem with the lunit number ( LUNs) becuase I think it is only allow to have 1-99.
> But I have over 700 files... Any idea what to do then?
Oh, dear! You probably only need two of these, at most. One to read the
data files (closing it after reading each file) and one to write the
results to the output file.
Something like this:
outLun = Get_Lun()
FOR j=0,numFiles-1 DO BEGIN
thisFile = files[j]
OpenR, inLun, thisFile, /Get_Lun
ReadF, inLun, ....
Free_Lun, inLun
... whatever you do to manipulate the data here
PrintF, outLun, manipulatedData
ENDFOR
Free_Lun, outLun
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Read several files into one big data array? [message #89513 is a reply to message #89512] |
Wed, 22 October 2014 15:52 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
David Fanning writes:
> Something like this:
>
> outLun = Get_Lun()
> FOR j=0,numFiles-1 DO BEGIN
> thisFile = files[j]
> OpenR, inLun, thisFile, /Get_Lun
> ReadF, inLun, ....
> Free_Lun, inLun
> ... whatever you do to manipulate the data here
> PrintF, outLun, manipulatedData
> ENDFOR
> Free_Lun, outLun
Whoops! Forgot to open that output file!
outLun = Get_Lun()
OpenW, outLun, outputfileName
FOR j=0,numFiles-1 DO BEGIN
thisFile = files[j]
OpenR, inLun, thisFile, /Get_Lun
ReadF, inLun, ....
Free_Lun, inLun
... whatever you do to manipulate the data here
PrintF, outLun, manipulatedData
ENDFOR
Free_Lun, outLun
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thue. ("Perhaps thou speakest truth.")
|
|
|
|
|
|
| Re: Read several files into one big data array? [message #89519 is a reply to message #89514] |
Wed, 22 October 2014 16:31 |
lucesmm
Messages: 26
Registered: October 2014
|
Junior Member |
|
|
On Wednesday, October 22, 2014 4:05:43 PM UTC-7, Craig Markwardt wrote:
> On Wednesday, October 22, 2014 4:57:12 PM UTC-4, luc...@gmail.com wrote:
>> I have tons of data files in the same folder. They are named correctly with the last 5 digits representing the order
> ...
>> data = FLTARR(nlines)
>
> Don't you want a more complex data type there? You are reading at least four numbers (integer, time, two floating point), so either it should be FLTARR(4,nlines), or a structure.
>
> Craig
Yes! Craig, thank for catching that!
I was getting only the first column
|
|
|
|
| Re: Read several files into one big data array? [message #89520 is a reply to message #89513] |
Wed, 22 October 2014 16:31 |
lucesmm
Messages: 26
Registered: October 2014
|
Junior Member |
|
|
On Wednesday, October 22, 2014 3:52:10 PM UTC-7, David Fanning wrote:
> David Fanning writes:
>
>> Something like this:
>>
>> outLun = Get_Lun()
>> FOR j=0,numFiles-1 DO BEGIN
>> thisFile = files[j]
>> OpenR, inLun, thisFile, /Get_Lun
>> ReadF, inLun, ....
>> Free_Lun, inLun
>> ... whatever you do to manipulate the data here
>> PrintF, outLun, manipulatedData
>> ENDFOR
>> Free_Lun, outLun
>
> Whoops! Forgot to open that output file!
>
> outLun = Get_Lun()
> OpenW, outLun, outputfileName
> FOR j=0,numFiles-1 DO BEGIN
> thisFile = files[j]
> OpenR, inLun, thisFile, /Get_Lun
> ReadF, inLun, ....
> Free_Lun, inLun
> ... whatever you do to manipulate the data here
> PrintF, outLun, manipulatedData
> ENDFOR
> Free_Lun, outLun
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
> Sepore ma de ni thue. ("Perhaps thou speakest truth.")
David, Thanks a lot!!!!
it works, now I have a huge matrix in a separate file :D
|
|
|
|
| Re: Read several files into one big data array? [message #89523 is a reply to message #89519] |
Thu, 23 October 2014 00:46 |
Mats Löfdahl
Messages: 263
Registered: January 2012
|
Senior Member |
|
|
Den torsdagen den 23:e oktober 2014 kl. 01:31:12 UTC+2 skrev luc...@gmail.com:
> On Wednesday, October 22, 2014 4:05:43 PM UTC-7, Craig Markwardt wrote:
>> On Wednesday, October 22, 2014 4:57:12 PM UTC-4, luc...@gmail.com wrote:
>>> I have tons of data files in the same folder. They are named correctly with the last 5 digits representing the order
>> ...
>>> data = FLTARR(nlines)
>>
>> Don't you want a more complex data type there? You are reading at least four numbers (integer, time, two floating point), so either it should be FLTARR(4,nlines), or a structure.
>>
>> Craig
>
> Yes! Craig, thank for catching that!
>
> I was getting only the first column
If you are just reading those files in order to put the contents in another file, why bother parsing the lines and putting them in an array?
It should be enough to just read each line as a string, add the file number to the beginning of that string, and then write it to the output file.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
