| Input Files - Size Limit? [message #8977] |
Mon, 19 May 1997 00:00  |
Mark Harvey
Messages: 1
Registered: May 1997
|
Junior Member |
|
|
Hi. I'm pretty much an IDL rookie but have been handed quite
a big project - developing an IDL-based analysis and
visualisation package for meteorological data.
My question is this: can IDL handle extremely large files (in
various data formats) without crashing out? I'm thinking of
sizes in the region of 100mb+.
Thanks for any help.
Regards,
Mark
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~- ~-~-~-~-~-~-
Mark Harvey
CRC Southern Hemisphere Meteorology 3rd Floor,Building 70
voice: +61-3-9905 9685 Monash University
fax : +61-3-9905 9689 Victoria 3168
email: mark@vortex.shm.monash.edu.au Australia
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~- ~-~-~-~-~-~-
|
|
|
|
| Re: Input Files - Size Limit? [message #8978 is a reply to message #8977] |
Sun, 18 May 1997 00:00  |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
Mark Harvey <mark@vortex.shm.monash.edu.au> writes:
> Hi. I'm pretty much an IDL rookie but have been handed quite
> a big project - developing an IDL-based analysis and
> visualisation package for meteorological data.
>
> My question is this: can IDL handle extremely large files (in
> various data formats) without crashing out? I'm thinking of
> sizes in the region of 100mb+.
Sure, put it into the variable "a". :-)
Seriously, Mark, there is no special limitations on the size
of data inside of IDL, but there are some practical limitations.
For example, most system administrators limit the amount
of process memory any one process can have. You have to
pretty much hand over your first born child to get them to
give you more.
Large files are usually not a problem with IDL, as long
as you don't read all the data into memory at once. Use
the associated variable method to read just that portion
of the data you need at any one time, and practice good
memory management techniques while you are working
with data. In practice, this means deleting or undefining
variables when you are done with them, etc. (See my
web page for ideas about this.) In IDL 5.0 you will
want to take advantage of pointers for sure. Just be
sure you delete the pointer data when you are done
with it.
If you run into trouble, there are lots of good IDL
programmers nearby. Justin Baker and Robert Dahni
at the Bureau of Meteorology there in Melbourne are
two names that come immediately to mind.
Oh, and ask for a machine that has *LOTS* of RAM!
Best of luck!
David
----------------------------------------------------------
David Fanning, Ph.D.
Fanning Software Consulting
Customizable IDL Programming Courses
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com
|
|
|
|
| Re: Input Files - Size Limit? [message #9026 is a reply to message #8977] |
Fri, 23 May 1997 00:00 |
pit
Messages: 92
Registered: January 1996
|
Member |
|
|
In article <5lv0b4$4it$1@news.dal.ca>,
sritcey@is.dal.ca (Stephen Ritcey) writes:
> When I first read David's message I assumed that he meant using
> something like
>
> a = assoc (unit, fltarr (a_not_impossibly_large_number), $
> offset = get_me_where_I_want_to_be)
>
> And now 'walk' through the file, repeatedly re-assoc-ing with larger and
> larger offsets. Now I agree that doing this 'manually' is a pain, but it
> is workable if the format of the data file is such that it make sense to
> take 'chunks' like this.
Not at all that complicated :-)
Just use
a = assoc (unit, fltarr (a_not_impossibly_large_number))
and use a as an array:
first=a(0)
second=a(1)
etc. I use this in my Speckle Code.
Peter
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Peter "Pit" Suetterlin http://www.uni-sw.gwdg.de/~pit
Universitaets-Sternwarte Goettingen
Tel.: +49 551 39-5048 pit@uni-sw.gwdg.de
-- * -- * ...-- * -- * ...-- * -- * ...-- * -- * ...-- * -- * ...-- * --
Come and see the stars! http://www.kis.uni-freiburg.de/~ps/SFB
Sternfreunde Breisgau e.V. Tel.: +49 7641 3492
____________________________________________________________ ______________
|
|
|
|
| Re: Input Files - Size Limit? [message #9040 is a reply to message #8977] |
Wed, 21 May 1997 00:00 |
haimov
Messages: 9
Registered: August 1996
|
Junior Member |
|
|
In article <337FA875.41C6@vortex.shm.monash.edu.au>,
mark@vortex.shm.monash.edu.au says...
>
> My question is this: can IDL handle extremely large files (in
> various data formats) without crashing out? I'm thinking of
> sizes in the region of 100mb+.
>
Mark,
I routinely work in IDL with files greater than 100 MB.
Once in a while I have to process files bigger than 1GB.
I have a dedicated workstation for this processing, but
my total memory is 128 M. The simplest solution for me
was to limit the chunk of data loaded into the memory
and use a loop to go through the whole file.
Sam
****************************
Samuel Haimov
Atmos. Science Dept.
University of Wyoming
tel: 1-307-766-2726
fax: 1-307-766-2635
email: haimov@uwyo.edu
****************************
|
|
|
|
| Re: Input Files - Size Limit? [message #9046 is a reply to message #8977] |
Wed, 21 May 1997 00:00 |
sritcey
Messages: 5
Registered: March 1997
|
Junior Member |
|
|
Mark Harvey:
: Can IDL handle extremely large files?
David Fanning:
> Large files are usually not a problem with IDL, as long
> as you don't read all the data into memory at once. Use
> the associated variable method to read just that portion
> of the data you need at any one time
Christian Marquardt:
: But: assoc is called with an [...]
: 'array_structure'. Thus, to assoc() my 120MB array stored
: in 'foo.dat', I'd use something like
: openr, unit, 'foo.dat', /get_lun
: a = assoc(unit, fltarr(very_large_number_of_elements))
: ...
: free_lun, unit
: Now, the IDL interpreter will first create the fltarr(...),
: and then pass the very large array over to the assoc function.
: The problem: if the fltarr(...) is larger than the (virtual)
: memory of my machine, IDL will not assoc anything but stop
: with a warning saying that there is not enough memory (I know -
: I tried...).
David has already replied, agreeing that it's a problem, but not
mentioning the following workaround:
When I first read David's message I assumed that he meant using
something like
a = assoc (unit, fltarr (a_not_impossibly_large_number), $
offset = get_me_where_I_want_to_be)
And now 'walk' through the file, repeatedly re-assoc-ing with larger and
larger offsets. Now I agree that doing this 'manually' is a pain, but it
is workable if the format of the data file is such that it make sense to
take 'chunks' like this.
[PVWave, not IDL: does it matter?]
|
|
|
|
| Re: Input Files - Size Limit? [message #9048 is a reply to message #8977] |
Wed, 21 May 1997 00:00 |
marq
Messages: 12
Registered: February 1996
|
Junior Member |
|
|
Hi David,
in your response to Mark you wrote:
> Mark Harvey <mark@vortex.shm.monash.edu.au> writes:
>
>> Hi. I'm pretty much an IDL rookie but have been handed quite
>> a big project - developing an IDL-based analysis and
>> visualisation package for meteorological data.
>>
>> My question is this: can IDL handle extremely large files (in
>> various data formats) without crashing out? I'm thinking of
>> sizes in the region of 100mb+.
>
> Sure, put it into the variable "a". :-)
>
> Large files are usually not a problem with IDL, as long
> as you don't read all the data into memory at once. Use
> the associated variable method to read just that portion
> of the data you need at any one time, and practice good
Hmm. You say one should use assoc? But: assoc is called
with an (as the documentation of IDL 5.6 beta 6 calls it)
'array_structure'. Thus, to assoc() my 120MB array stored
in 'foo.dat', I'd use something like
openr, unit, 'foo.dat', /get_lun
a = assoc(unit, fltarr(very_large_number_of_elements))
...
free_lun, unit
Now, the IDL interpreter will first create the fltarr(...),
and then pass the very large array over to the assoc function.
The problem: if the fltarr(...) is larger than the (virtual)
memory of my machine, IDL will not assoc anything but stop
with a warning saying that there is not enough memory (I know -
I tried...).
Thus: it seems to me that assoc can be used only if the
data stored in the file would fit into the memory. Or to
make a question out of it: is there a way to use assoc()
_without_ letting IDL trying to allocate all the memory at once?
Thanks a lot...
> Oh, and ask for a machine that has *LOTS* of RAM!
Or is that the only solution left?
Regards from Berlin,
Chris.
------------------------------------------------------------ ---------------
Christian Marquardt
Meteorologisches Institut der | tel.: (+49) 30-838-71170
Freien Universitaet Berlin | fax.: (+49) 30-838-71167
Carl-Heinrich-Becker-Weg 6-10 | email: marq@strat01.met.fu-berlin.de
D-12165 Berlin |
------------------------------------------------------------ ---------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
