comp.lang.idl-pvwave archive: archive » Handle big data files

Home » Public Forums » archive » Handle big data files

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Handle big data files [message #92223 is a reply to message #92221]

Mon, 02 November 2015 01:13

Helder Marchetto
Messages: 520
Registered: November 2011

Senior Member

Hi,
I think that this line is responsible for making things slow:

data=[[data],[line]]

If the array data gets to be lon, then it will take a long time to copy the previous data to a new variable and add one element...

You have two options:
1) only valid for IDL version >8.0. Use a list(). before the for use:
data = list()
then instead of data=[[data],[line]] use:
data->add, line
Then at the end:
PrintF, outLun, data->toArray(), FORMAT= '...'

2) it's more complicated, but general. Create the data array loooong, then fill it up. You could also actually guess it's length:
nData = 0l
FOR i=0,nfiles-1 DO BEGIN
nlines = FILE_LINES(files[i])
nData += nlines-1 ;one line you always disregard
ENDFOR

Now create data so that it is long enough:
myDataStructure = make_array(17,1, type=5)
data = replicate(myDataStructure, nData)

and in the cycle you fill up. You will also need a "fill-up" counter:

fillCounter = 0l
FOR...
...
while...
...
data[fillCounter] = line
fillCounter++
endwhile
...
endfor

I hope it helps...

Cheers,
Helder

Report message to a moderator

[Message index]

		Handle big data files By: lucesmm on Sun, 01 November 2015 17:44
		Re: Handle big data files By: Helder Marchetto on Mon, 02 November 2015 01:13
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 07:02
		Re: Handle big data files By: Helder Marchetto on Mon, 02 November 2015 08:26
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 08:36
		Re: Handle big data files By: lucesmm on Mon, 02 November 2015 09:03

Previous Topic:	Read data from file
Next Topic:	not an IDL question.... scrolling in nedit

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 15:28:34 PDT 2025

Total time taken to generate the page: 0.00494 seconds