| Re: Memory Headaches [message #31575] |
Fri, 02 August 2002 07:03  |
crono15m
Messages: 9
Registered: June 2002
|
Junior Member |
|
|
Thank you everyone who replied.
I would love to add more RAM to my system. Unfortunately I am limited
at 512 by my motherboard.
I recieved the following e-mail from an RSI employee who wishes to
remain nameless.
>> Ben,
>> If you *were* to get VARRAY to work under Windows (and you
>> could, if you were willing to learn some Win32 API), you would find
>> that it does not help you escape your virtual memory limits.
>> Memory mapped files count against your VM just as certainly as
>> any other memory does. If you consider the matter, you'll realize
>> that if it's in your address space, it must be consuming virtual
memory.
>> Furthermore, if the OS chooses to map the file in the middle of your
>> address space (and some do), it can cause memory fragmentation,
which
>> can cause other problems. Hence, memory mapping, though useful, is
>> hardly a silver bullet against memory limits.
>> The only difference between memory mapped files and "regular" memory
is
>> that "regular" memory is backed by your swap space, while mapped
files
>> are backed by the file itself. A very minor difference really ---
it's
>> disk space in either case, and virtual memory is virtual memory no
matter
>> where it is backed.
>> There is one exception to the above: If your system does not have
>> a large enough swap space, then you should immediately proceed to
>> find out how to increase it. There's no reason not to have an
enormous
>> swap space given how cheap disk is now.
>> I can assure you that IDL does not cap the amount of memory you
>> can allocate. If you are running out of memory, it is because the
>> OS is refusing to give it more. Sometimes you can increase the
>> limit if you know your OS well enough. Sometimes, you can't
>> (for instance, WinNT limits process memory at 1GB).
>> If you're stuck on Windows, and you can't increase the memory
available,
>> then you'll have to find a way to process the data in smaller chunks
>> instead. Or perhaps you can find a more efficient way to express the
>> computation that will require less memory.
>> Sorry about the bad news, but I hope this helps... If nothing else,
>> I've saved you the time pursuing memory mapping to solve this
problem.
I have a few things to say about this. First of all I appreciate RSI
taking an interest in helping and that I got two e-mails from RSI
shortly after my original message was posted.
Second, I ran a few tests.
I did the following at the command line.
a = intarr(reallybignumber)
and found what the biggest number was that for which IDL would succeed
in allocating memory. (This took a while)
On the first system I tested with (512MB Ram and 768MB swap) the
largest number was a = intarr(408846274l)
The system memory usage stats at that point were 890660KB Used/
1279356KB Limit
with peak usage at 958236KB
I then increased the amount of virtual memory...
Largest: a = intarr(408846274l)
System stats: 884456KB Used / 1803648KB Limit Peak: 886952
At this point I thought the two possibilities were fragmentation or OS
limit.
So, I did the same thing on an identical machine (we have two machines
with the exact same hardware setup)
The largest number was exactly the same on the second system. To me,
this rules out fragmentation (short of a huge coincidence).
Both of the test computers are running Win2k.
I then ran the same test on a machine running NT4 Sp6 (512MB ram)
The largest array on that system was around 528,000,000l independent
of how high I set the swap. (assuming the swap was big enough)
This makes me think that the limit probably is OS dependant.
I have yet to test a memory mapping program though to see if this
would increase the limit but I'm not yet convinced that trying memory
mapping would be a waste of time.
Ben
|
|
|
|
| Re: Memory Headaches [message #31579 is a reply to message #31575] |
Thu, 01 August 2002 17:01  |
Pete[1]
Messages: 7
Registered: April 1999
|
Junior Member |
|
|
Hi Ben,
As far as VARRAY-like support for Win32 is concerned, you have two options
that I know about.
The first is in Randall Frank's "IDL tools" DLM, which you can find on Ronn
Kling's website (http://rlkling.com/) under Freeware. Amongst other
things, this DLM contains functions for memory-mapped-file support.
The second is a DLM I've written myself. I was going to post something
about it to the group earlier, but changed my mind when I checked the web
(shortly before finishing my DLM) and discovered that Randall Frank had
already done MMF support. As the issue has come up again, and as my DLM
has a bit of a different slant to it, I've changed my mind again, so here it
is: Binary at ftp://ftp.syd.dem.csiro.au/stoneface.zip and source at
ftp://ftp.syd.dem.csiro.au/stoneface_src.zip. Unfortunately this FTP
server isn't under my control and it has some peculiar ideas about access
rights. If you have any problems with the FTP, email me and I'll email you
the ZIP.
About my DLM: It's for Win32 only. I've used it on WinNT4, but it should
be okay for Win2000 and maybe even Win9x. It has quite a bit of
flexibility on the type of view to be created (array type and dimensions,
even structure arrays), it can create more than one (coherent) view on a
file at a time, and it can quickly change something (the type, size or
offset) of a view - useful for performing semi-random access on huge (multi
gigabyte) files. The documentation includes some blurb about your enemy,
the Win32 adress space.
Cheers
Peter Mason
CSIRO Division of Exploration and Mining
Sydney, Australia
|
|
|
|
|
|
|
|
| Re: Memory Headaches [message #31715 is a reply to message #31575] |
Fri, 02 August 2002 17:42 |
Mark Rivers
Messages: 49
Registered: February 2000
|
Member |
|
|
Ben <crono15m@aol.com> wrote in message
news:fbb3dcd9.0208020603.34689f8f@posting.google.com...
> Thank you everyone who replied.
>
> I would love to add more RAM to my system. Unfortunately I am limited
> at 512 by my motherboard.
>
> I then ran the same test on a machine running NT4 Sp6 (512MB ram)
>
> The largest array on that system was around 528,000,000l independent
> of how high I set the swap. (assuming the swap was big enough)
>
> This makes me think that the limit probably is OS dependant.
This is consistent with my experience with Windows NT. I have 2 machines
with 1 GB of RAM and 3GB of swap space each, and the most memory IDL can
allocate is 1GB. This is also what RSI also told you was the NT limit.
Note however, that this is inconsistent with every piece of Windows NT
documentation I have found, which states that a single NT process can
allocate 2GB (not 1GB). In my case the difference would be very
significant, and if IDL could access 2GB I would go buy a new computer with
2GB of RAM in a heartbeat. (My application is 3-D tomography data sets,
where being able to quickly look at slices in any direction is critical, and
makes re-reading the desired subset data from disk totally impractical.
Of course, I have not yet taken the 10 minutes it would take to test the
following C program
main()
{
char *t = malloc(unsigned long(1024*1024*1024*1.5));
}
to see if a C program can allocate more than 1GB of memory as the NT
documentation indicates, or whether a C program is also limited to 1GB.
Just laziness!
Mark Rivers
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
