| Re: Input Files - Size Limit? [message #9048 is a reply to message #8977] |
Wed, 21 May 1997 00:00  |
marq
Messages: 12
Registered: February 1996
|
Junior Member |
|
|
Hi David,
in your response to Mark you wrote:
> Mark Harvey <mark@vortex.shm.monash.edu.au> writes:
>
>> Hi. I'm pretty much an IDL rookie but have been handed quite
>> a big project - developing an IDL-based analysis and
>> visualisation package for meteorological data.
>>
>> My question is this: can IDL handle extremely large files (in
>> various data formats) without crashing out? I'm thinking of
>> sizes in the region of 100mb+.
>
> Sure, put it into the variable "a". :-)
>
> Large files are usually not a problem with IDL, as long
> as you don't read all the data into memory at once. Use
> the associated variable method to read just that portion
> of the data you need at any one time, and practice good
Hmm. You say one should use assoc? But: assoc is called
with an (as the documentation of IDL 5.6 beta 6 calls it)
'array_structure'. Thus, to assoc() my 120MB array stored
in 'foo.dat', I'd use something like
openr, unit, 'foo.dat', /get_lun
a = assoc(unit, fltarr(very_large_number_of_elements))
...
free_lun, unit
Now, the IDL interpreter will first create the fltarr(...),
and then pass the very large array over to the assoc function.
The problem: if the fltarr(...) is larger than the (virtual)
memory of my machine, IDL will not assoc anything but stop
with a warning saying that there is not enough memory (I know -
I tried...).
Thus: it seems to me that assoc can be used only if the
data stored in the file would fit into the memory. Or to
make a question out of it: is there a way to use assoc()
_without_ letting IDL trying to allocate all the memory at once?
Thanks a lot...
> Oh, and ask for a machine that has *LOTS* of RAM!
Or is that the only solution left?
Regards from Berlin,
Chris.
------------------------------------------------------------ ---------------
Christian Marquardt
Meteorologisches Institut der | tel.: (+49) 30-838-71170
Freien Universitaet Berlin | fax.: (+49) 30-838-71167
Carl-Heinrich-Becker-Weg 6-10 | email: marq@strat01.met.fu-berlin.de
D-12165 Berlin |
------------------------------------------------------------ ---------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
