comp.lang.idl-pvwave archive: archive » Handle big data files

Home » Public Forums » archive » Handle big data files

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Handle big data files [message #92221]

Sun, 01 November 2015 17:44

lucesmm
Messages: 26
Registered: October 2014

Junior Member

Hello
I have been working with big data files (~18MB each)

And everytime I need to handle anything with the data, mostly plotting, takes a long time.
So I was trying to put them all in one file per year (the data is arrange by date)
But just running this takes a few hours.

PRO Save_1year
GET_LUN, outlun
Openw, outlun, 'name.txt'
Year = ''
READ, Year, PROMPT='Enter Year:'
openw, outlun, 'C:data\year'+ Year+'.txt', /get_lun

files=FILE_SEARCH('C:data'+Year+'*', COUNT=nfiles)
Print, files
data = []

FOR i=0,nfiles-1 DO BEGIN
filename=files[i]
fileNumber = STRMID(filename,51,5)
nlines = FILE_LINES(files[i])
thisFile = files[i]

;line=fltarr(17,1)
OpenR, inLun, thisFile, /Get_Lun
SKIP_LUN, inLun, 1, /LINES
while not EOF(inlun) do begin & $
line=make_array(17,1, type=5)
Readf, inLun, line, FORMAT='...'
data=[[data],[line]]

Endwhile
Free_Lun, inLun

ENDFOR
PrintF, outLun, data, FORMAT= '...'
END

I was wondering if there is an easier way to handle this data, I don't know anything about SAVE files, may be this is easier?
Please help

Thank you

Luz Maria

Report message to a moderator

[Message index]

		Handle big data files By: lucesmm on Sun, 01 November 2015 17:44
		Re: Handle big data files By: Helder Marchetto on Mon, 02 November 2015 01:13
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 07:02
		Re: Handle big data files By: Helder Marchetto on Mon, 02 November 2015 08:26
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 08:36
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 09:03

Previous Topic:	Read data from file
Next Topic:	not an IDL question.... scrolling in nedit

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 15:49:42 PDT 2025

Total time taken to generate the page: 0.00447 seconds