| Re: Getting memory overflow on array concat; wh [message #3490 is a reply to message #3433] |
Sat, 28 January 1995 05:35  |
knipp
Messages: 68
Registered: January 1993
|
Member |
|
|
In article r43@umd5.umd.edu, bleau@umdsp.umd.edu writes:
> Hello. I'm a fairly new user of IDL; so is a user of mine, on whose behalf I
> am posting this message. We are using IDL V2.0 on OpenVMS VAX.
>
> The task is to read in a set of numbers from a disk file into a 2-dimensional
> array. The disk file has 3 numbers per line, about 4000 lines. I include at
> the end of this post the IDL code which does the read. Problem is, the code
> never finishes; IDL runs out of memory beforehand. I increased the user's max
> memory allotment, and the program went a little further before running out of
> memory. Back-of-envelope calculations showed the user needs about 48Kb to hold
> this data, of 93 pages (512 bytes/page). IDL itself uses ~6000 pages. I gave
> her account an upper limit of 25000 pages; enough room for IDL and a 810666x3
> array! What is going on???
>
> I suspect the array concatenation operator is the culprit, but I can't say why.
> Here's the relevant code fragment, which is within a loop:
> READF, 1, temp
> IF first THEN BEGIN
> data = temp
> first = ''
> ENDIF ELSE BEGIN
> data = [data,temp]
> ENDELSE
> Both data and temp have been previously declared FLTARR(1,parameters), where
> parameters is 3. The help command, when given after the crash, shows DATA as
> being something like (3100,3), and only 2% of the symbol space used.
>
> As I said, this doesn't make sense. My hunch is there's a side effect of array
> concatenation that eating up memory, never releasing it. Has anyone has
> experience with this type of programming bug and how to fix it?
>
> Larry Bleau
> University of Maryland
> bleau@umdsp.umd.edu
> 301-405-6223
>
... (stuff deletet)
>
> Larry Bleau
> University of Maryland
> bleau@umdsp.umd.edu
> 301-405-6223
Your idea about "eating up memory, never releasing it" is to my understanding
correct. You may overcome your problem by
- define array DAT=FLTARR(1,n_lines)
- read in using a FOR - loop:
FOR i=0, n_lines-1l do
READF, unit, tmp
DAT(*,i) = tmp
ENDFOR
- if you do not know the number of lines in your data-file, and want IDL
to get that number
- open the file
- get the filesize via FSTAT
- define BFILE = bytarr(file-size)
- readu, unit, bytarr
- count number of linefeeds (pos = where(BFILE eq 10b))
n_lines = n_elements(pos) (if pos(0) ne -1)
- close file
- set BFILE=0 (relesing memory !!)
- define your data-array
Hop this helps
____________________________________________________________ __________________
__ ____ __
/ // _ \ / / Karlheinz Knipp phone: +49 511 - 762 4922
/ // /_/ // / University of Hannover fax: +49 511 - 762 2483
/ // ____// / Institute for Photogrammetry
/ // / / / Nienburger Str.1
/_//_/ /_/ FRG 30167 Hannover e-mail: knipp@ipi.uni-hannover.de
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
