| Re: Input Files - Size Limit? [message #8952] |
Tue, 20 May 1997 00:00 |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
Christian Marquardt questions one of my answers when he writes:
> Hmm. You say one should use assoc? But: assoc is called
> with an (as the documentation of IDL 5.6 beta 6 calls it)
> 'array_structure'. Thus, to assoc() my 120MB array stored
> in 'foo.dat', I'd use something like
>
> openr, unit, 'foo.dat', /get_lun
> a = assoc(unit, fltarr(very_large_number_of_elements))
> ...
> free_lun, unit
>
> Now, the IDL interpreter will first create the fltarr(...),
> and then pass the very large array over to the assoc function.
> The problem: if the fltarr(...) is larger than the (virtual)
> memory of my machine, IDL will not assoc anything but stop
> with a warning saying that there is not enough memory (I know -
> I tried...).
>
> Thus: it seems to me that assoc can be used only if the
> data stored in the file would fit into the memory. Or to
> make a question out of it: is there a way to use assoc()
> _without_ letting IDL trying to allocate all the memory at once?
Absolutely correct. One hopes that your 120MB data file is not
one array, but these days that assumption is being questioned over
and over again.
I have been saying for quite a while now that RSI needs to
develop some method of memory tiling (ala the method the
folks who developed ENVI have implemented) so that normal
people on normal machines will have the opportunity to work
with the really big datasets too. Tiling would allow you
to read and work with "chunks" of data instead of the
entire data set at once. These kind of numbers are getting
to be all too familiar with satellites up there collecting terrabytes
of data a day.
In this 120MB array situation, I would say your best bet
would be to give Achim Hein a call and see if you can borrow
his monster workstation with the 2 GByte of RAM for a day
or so. :-)
Regards,
David
----------------------------------------------------------
David Fanning, Ph.D.
Fanning Software Consulting
Customizable IDL Programming Courses
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
