| Re: Memory Headaches [message #31575] |
Fri, 02 August 2002 07:03  |
crono15m
Messages: 9
Registered: June 2002
|
Junior Member |
|
|
Thank you everyone who replied.
I would love to add more RAM to my system. Unfortunately I am limited
at 512 by my motherboard.
I recieved the following e-mail from an RSI employee who wishes to
remain nameless.
>> Ben,
>> If you *were* to get VARRAY to work under Windows (and you
>> could, if you were willing to learn some Win32 API), you would find
>> that it does not help you escape your virtual memory limits.
>> Memory mapped files count against your VM just as certainly as
>> any other memory does. If you consider the matter, you'll realize
>> that if it's in your address space, it must be consuming virtual
memory.
>> Furthermore, if the OS chooses to map the file in the middle of your
>> address space (and some do), it can cause memory fragmentation,
which
>> can cause other problems. Hence, memory mapping, though useful, is
>> hardly a silver bullet against memory limits.
>> The only difference between memory mapped files and "regular" memory
is
>> that "regular" memory is backed by your swap space, while mapped
files
>> are backed by the file itself. A very minor difference really ---
it's
>> disk space in either case, and virtual memory is virtual memory no
matter
>> where it is backed.
>> There is one exception to the above: If your system does not have
>> a large enough swap space, then you should immediately proceed to
>> find out how to increase it. There's no reason not to have an
enormous
>> swap space given how cheap disk is now.
>> I can assure you that IDL does not cap the amount of memory you
>> can allocate. If you are running out of memory, it is because the
>> OS is refusing to give it more. Sometimes you can increase the
>> limit if you know your OS well enough. Sometimes, you can't
>> (for instance, WinNT limits process memory at 1GB).
>> If you're stuck on Windows, and you can't increase the memory
available,
>> then you'll have to find a way to process the data in smaller chunks
>> instead. Or perhaps you can find a more efficient way to express the
>> computation that will require less memory.
>> Sorry about the bad news, but I hope this helps... If nothing else,
>> I've saved you the time pursuing memory mapping to solve this
problem.
I have a few things to say about this. First of all I appreciate RSI
taking an interest in helping and that I got two e-mails from RSI
shortly after my original message was posted.
Second, I ran a few tests.
I did the following at the command line.
a = intarr(reallybignumber)
and found what the biggest number was that for which IDL would succeed
in allocating memory. (This took a while)
On the first system I tested with (512MB Ram and 768MB swap) the
largest number was a = intarr(408846274l)
The system memory usage stats at that point were 890660KB Used/
1279356KB Limit
with peak usage at 958236KB
I then increased the amount of virtual memory...
Largest: a = intarr(408846274l)
System stats: 884456KB Used / 1803648KB Limit Peak: 886952
At this point I thought the two possibilities were fragmentation or OS
limit.
So, I did the same thing on an identical machine (we have two machines
with the exact same hardware setup)
The largest number was exactly the same on the second system. To me,
this rules out fragmentation (short of a huge coincidence).
Both of the test computers are running Win2k.
I then ran the same test on a machine running NT4 Sp6 (512MB ram)
The largest array on that system was around 528,000,000l independent
of how high I set the swap. (assuming the swap was big enough)
This makes me think that the limit probably is OS dependant.
I have yet to test a memory mapping program though to see if this
would increase the limit but I'm not yet convinced that trying memory
mapping would be a waste of time.
Ben
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
