| Re: memory issues redux [message #41566] |
Tue, 16 November 2004 15:23  |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"Karl Schultz" <kschultz_no_spam@rsinc.com> wrote in message
news:10pl0rlb7q75720@corp.supernews.com...
> "R.G.Stockwell" <noemail@please.com> wrote in message
> news:2vv1edF2n9tf7U1@uni-berlin.de...
>> Thanks for all the info Karl!
>> If I make any progress with editbin and /3gb I'll post about it.
>
> We managed to find a machine here that would boot with /3GB. We did the
> EDITBIN trick on idlde.exe.
...
>
> Hope this helps,
> Karl
Again, thanks a lot Karl. Great information, I'm reading through those
links.
Cheers,
bob
--
In order to bend the ram first you must know that there is no ram ;)
|
|
|
|
| Re: memory issues redux [message #41567 is a reply to message #41566] |
Tue, 16 November 2004 14:51  |
Karl Schultz
Messages: 341
Registered: October 1999
|
Senior Member |
|
|
"R.G.Stockwell" <noemail@please.com> wrote in message
news:2vv1edF2n9tf7U1@uni-berlin.de...
> Thanks for all the info Karl!
> If I make any progress with editbin and /3gb I'll post about it.
We managed to find a machine here that would boot with /3GB. We did the
EDITBIN trick on idlde.exe.
The result is that we were not able to allocate an array any larger than we
were able to in the 2G environment. The reason is that the system modules
are still loaded just below the 2G line and that fragments the virtual
address space. So the limit on the largest contiguous block of memory
doesn't change much.
We were, however, able to allocate a total of 2.8 GB by allocating many
smaller arrays, working down to about 5MB. Now, this doesn't really help
applications that are designed to work only with large arrays. But it does
show that the /3GB switch seems to work and setting LARGEADDRESSAWARE with
editbin appears to work. But again, if you want to allocate two 1.3 GB
arrays, as you said in the original posting, I don't think that the 3GB
environment is likely to help you much. And it won't help other people who
think that the 3GB environment is going to solve certain memory problems.
The answer is, and always will be, change your app to work with smaller
chunks, or go 64-bit. The 3GB environment just gives you more smaller
chunks.
Another thing to keep in mind is that booting with 3GB is not free. The
extra 1G of virtual address space must come from someplace, and in this case
it is the operating system, since the upper 2G had previously been reserved
for it. This means that certain system resources like file caches, system
page tables, GDI bitmap storage space, etc, now have less space. So,
turning on 3GB may also have its disadvantages. I believe that the original
intent behind the 3GB switch was to let large database applications run on
"server" versions of Windows. These systems are carefully tuned to run as
servers and just run the DB application. They were not set up to run
interactive things like IDL. Bottom line is that the 3GB switch may not be
for everybody. I suppose I'd suggest that people need to fully understand
the memory issues before investing a lot of time in this. It may be better
in the long run to target a more capable OS. These short term adjustments
that provide a sort of help that isn't what is really needed might not be a
good investment.
I hesitate to point to this link because I don't know how authoritative it
is, but many of the postings that I did read seem to explain what we are
talking about. There are actually some good tutorial-style discussions
about virtual memory and the 3GB issue.
http://weblogs.asp.net/oldnewthing/archive/2004/08/22/218527 .aspx
Microsoft explains a problem with using the 3GB switch with XP SP1 in
http://support.microsoft.com/default.aspx?scid=kb;en-us;328269
And of course, there is much more out there on the net and in good books
about Windows.
Hope this helps,
Karl
|
|
|
|
|
|
| Re: memory issues redux [message #41573 is a reply to message #41571] |
Tue, 16 November 2004 10:33 |
Karl Schultz
Messages: 341
Registered: October 1999
|
Senior Member |
|
|
"R.G.Stockwell" <noemail@please.com> wrote in message
news:2vspuaF2ocjacU1@uni-berlin.de...
> I'm trying to squeeze out as much of my ram as I can.
> The threads here have helped a lot, but I still have a couple issues
> and questions:
> [win xp pro sp2, 3.4ghz p4, 4gb ram]
>
> 1) I'm not clear as to the status of idlde being able to access the 3gb
> memory
> space (by changing the boot ini file to include a /3g command).
> Can v6.11 do that? Is idl "Large Address Space Aware" ?
No, no current version of IDL (including 6.1.1) is Large Address Space
Aware.
But see additional discussion later on in this post.
> If I could get this working, that would be fantastic. I have not tried
> messing around with my boot ini file yet.
If you do this, be extremely careful.
Don't just add the /3GB switch to the currently active OS.
Copy the active entry and add the /3GB switch. Then you'll get a choice at
boot time.
That part of my boot.ini looks like this:
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP
Professional" /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP
Professional" /fastdetect /3GB
C:\bootfc1n.lnx="Fedora Core 1 (Yarrow)"
On my machine, I got an immediate blue screen in 3GB mode. So, I was glad I
made it an option on the boot menu.
I strongly suspect that one my drivers, probably the video driver, is the
problem.
When I get a chance, I can try to boot in VGA mode and/or try a different
video card. But I'm not optimistic because it is likely that another driver
might have a problem. There have been quite a few reports on the internet
about the difficulty in getting /3G working and its poor stability when it
does. But if I get anywhere, I'll let you know.
> 2) I used the editbin program to rebase the dlls, and saw no difference
> in the largest array possible. My best is a whimpy 940mb array.
> Is there any way to figure out what is going on in my ram, to see what
> dlls are loaded where, etc? Anyone know of a program that can defrag ram?
> I've googled and downloaded several ram defragers, but they don't have any
> effect on memory (in fact, they all see that I have 2gb of ram, and that
> I'm using
> 0kb if it). I also came accros a tech article saying that these types of
> programs
> are just a scam. So anyone know of a real program to manage ram? Or at
> least
> look at the ram to see what is loaded where?
I loaded IDL with the MS Visual Studio and looked at the module locations
after starting the IDLDE:
ug3220.dll 00220000-0023B000
MesaGLU6_2.dll 00240000-0028F000
MesaGL6_2.dll 00290000-003F4000
idlde.exe 00400000-005DB000
osmesa6_2.dll 009D0000-009DA000
freetype2_1_3.dll 009E0000-00A2C000
msvcr70d.dll 00A30000-00AB5000
LMAAG2DA.DLL 01760000-017B0000
wingl32.dll 017B0000-017E0000
idl32.dll 10000000-10794000
-- 1 GB gap --
shell32.dll 4F510000-4FD21000
ddraw.dll 51000000-51047000
msvcp60.dll 55900000-55961000
uxtheme.dll 5AD70000-5ADA4000
mfc70enu.dll 5D360000-5D36E000
opengl32.dll 5ED00000-5EDC6000
glu32.dll 68B20000-68B3E000
SHLWAPI.DLL 70A70000-70AD9000
comctl32.dll 71950000-71A34000
ws2help.dll 71AA0000-71AA8000
ws2_32.dll 71AB0000-71AC5000
netapi32.dll 71C20000-71C6E000
winspool.drv 73000000-73023000
dciman32.dll 73BC0000-73BC6000
icmp.dll 74290000-74294000
oleacc.dll 74C80000-74CAC000
riched20.dll 74E30000-74E9A000
comdlg32.dll 763B0000-763F5000
iphlpapi.dll 76D60000-76D77000
secur32.dll 76F90000-76FA0000
oleaut32.dll 77120000-771AB000
ole32.dll 771B0000-772D4000
comctl32.dll 77340000-773CB000
version.dll 77C00000-77C07000
msvcrt.dll 77C10000-77C63000
user32.dll 77D40000-77DCC000
advapi32.dll 77DD0000-77E5D000
kernel32.dll 77E60000-77F46000
ntdll.dll 77F50000-77FF7000
rpcrt4.dll 78000000-78087000
msvcr70.dll 7C000000-7C054000
mfc70d.dll 7C140000-7C31C000
gdi32.dll 7F000000-7F041000
Now these are just the modules and there are certainly other things using
this memory. Just because there are gaps between modules doesn't mean that
anything is wrong. But there is an open area of just over 1G, which is
about the size of your best allocation. So, it sort of looks like you are
getting a pretty nominal result.
I know I had a really cool tool on an old NT machine that let me look at the
memory layout. I can't find it and I look for it again everytime this comes
up. But no, I don't think that there are RAM (actually virtual address)
defraggers. Such a program would have to know about every single reference
to a virtual address if something at that address got moved, and that's just
not practical in an OS like Windows.
This is one area where the 32-bit and 64-bit Unix's have an edge and this is
why people use them.
However,
If you do get /3GB to work on your machine, then you can modify the
idlde.exe file with the EDITBIN tool to turn on the Large Address Space
Aware flag and see what happens. (EDITBIN comes with MS developer tools
like Visual Studio). Of course I have to say that RSI doesn't support this
and I can't tell you what will happen. We do not turn this flag on be
default but we'll definitely consider it for future releases.
As pointed out in other messages in this thread, 32-bit IDL cannot allocate
a contiguous piece of memory larger than 2GB. It would take an enormous
amount of work to get into the 2GB-4GB zone with 32-bit IDL and one can
argue it isn't worth it when 64-bit OS's are available. 64-bit Linux is here
and 64-bit Windows is almost here.
So, getting the 3GB and Large Address Space Aware configuration going won't
let you make single arrays over 2GB in size, but it will let you make more
arrays of this size or smaller. Also, a 32-bit LASA application will likely
have access to 4GB of space when running on 64-bit Windows.
Hope this helps,
Karl
|
|
|
|
| Re: memory issues redux [message #41574 is a reply to message #41573] |
Tue, 16 November 2004 10:08 |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
"Paolo Grigis" <pgrigis@astro.phys.ethz.ch> wrote in message
news:419a3d56$1@news1.ethz.ch...
> Actually, you can put a smaller number of, say, floats in
> an array than bytes
>
> IDL> a=fltarr(1024L*1024*511) ;this is ok
> IDL> a=fltarr(1024L*1024*512)
> % Array has too many elements.
> % Execution halted at: $MAIN$
>
> So it looks like IDL internally adresses the memory in byte sized
> units, indipendently from the type of data stored, and therefore
> you can get up to 2 gigaelements for a byte array, but just 512
> megaelements for a float array, 256 megaelements for a double
> array etc.
>
> Therefore the real limit for a single array is at 2GB, no matter
> the array type.
>
> Cheers,
> Paolo
I'm a bit surprised to see that with floats. I was suspecting that
complex variables were just 2 floats internally, not that everything was
bytes. huh!
cheers,
bob
|
|
|
|
|
|
| Re: memory issues redux [message #41576 is a reply to message #41575] |
Tue, 16 November 2004 09:49 |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
Actually, you can put a smaller number of, say, floats in
an array than bytes
IDL> a=fltarr(1024L*1024*511) ;this is ok
IDL> a=fltarr(1024L*1024*512)
% Array has too many elements.
% Execution halted at: $MAIN$
So it looks like IDL internally adresses the memory in byte sized
units, indipendently from the type of data stored, and therefore
you can get up to 2 gigaelements for a byte array, but just 512
megaelements for a float array, 256 megaelements for a double
array etc.
Therefore the real limit for a single array is at 2GB, no matter
the array type.
Cheers,
Paolo
R.G.Stockwell wrote:
> Marc, Paolo,
> thanks for the responses.
>
> Wow, that is much much better ram management under linux.
>
> I'll be using complex arrays actually, so the "not enough elements"
> should not be a problem (in principle, who knows what goes on in
> the bowels of idl).
>
> Looks like the thing to do would be to dual boot my computer
> and move idl over to the linux part. (too bad, because I am a
> big fan of idlde under windows, and I'm not a emacs guy. But
> i guess a scientist's gotta do what a scientist's gotta do).
>
> thanks,
> bob
>
>
--
____________________________________________________________ ________
Paolo Grigis
ETHZ - Institute of Astronomy email: pgrigis@astro.phys.ethz.ch
Scheuchzerstrasse 7
ETH Zentrum phone: ++41 1 632 42 20
8092 Zurich fax : ++41 1 632 12 05
Switzerland http://www.astro.phys.ethz.ch/
____________________________________________________________ ________
------------ And now a word from our sponsor ------------------
For a quality usenet news server, try DNEWS, easy to install,
fast, efficient and reliable. For home servers or carrier class
installations with millions of users it will allow you to grow!
---- See http://netwinsite.com/sponsor/sponsor_dnews.htm ----
|
|
|
|
|
|
| Re: memory issues redux [message #41579 is a reply to message #41577] |
Tue, 16 November 2004 03:11 |
Paolo Grigis
Messages: 171
Registered: December 2003
|
Senior Member |
|
|
R.G.Stockwell wrote:
> I'm trying to squeeze out as much of my ram as I can.
> The threads here have helped a lot, but I still have a couple issues
> and questions:
> [win xp pro sp2, 3.4ghz p4, 4gb ram]
>
> 1) I'm not clear as to the status of idlde being able to access the 3gb
> memory
> space (by changing the boot ini file to include a /3g command).
> Can v6.11 do that? Is idl "Large Address Space Aware" ?
> If I could get this working, that would be fantastic. I have not tried
> messing around with my boot ini file yet.
>
> 2) I used the editbin program to rebase the dlls, and saw no difference
> in the largest array possible. My best is a whimpy 940mb array.
> Is there any way to figure out what is going on in my ram, to see what
> dlls are loaded where, etc? Anyone know of a program that can defrag ram?
> I've googled and downloaded several ram defragers, but they don't have any
> effect on memory (in fact, they all see that I have 2gb of ram, and that
> I'm using
> 0kb if it). I also came accros a tech article saying that these types of
> programs
> are just a scam. So anyone know of a real program to manage ram? Or at
> least
> look at the ram to see what is loaded where?
>
> 3) how are idl memory limitations under linux? Is it limited to 2gb, or
> could
> one access more memory than that? I am interested in both (maximum array
> size, and total memory available), if I could have 2 arrays of 1.3gb each,
> that
> would be awesome!
Under Linux (32 bit), for a single array you can get as large as
2 GB (minus a couple of overhead bytes):
IDL> a=bytarr(1024L*1024*2047+1024L*1023+1014L);works
IDL> a=bytarr(1024L*1024*2047+1024L*1023+1015L)
% Array has too many elements.
% Execution halted at: $MAIN$
Which is due to the limitation of the indexing occurring with longs,
and therefore not being able to go beyond 0..2^31-1.
As for having more than one array, I was just able to allocate for
a second array b fewer memory than a, and for a third array c fewer memory
than b etc. Probably has to do with fragmentation of the available RAM (?).
Anyway I could get something like (d created after c, c after b, b after a, and
trying to get as much as possible):
IDL> help,a,b,c,d
A BYTE = Array[2147483638]
B BYTE = Array[465567744]
C BYTE = Array[232783872]
D BYTE = Array[143654912]
IDL> help,/mem
heap memory used: 2989922225, max: 3080099849, gets: 415, frees:
At least there is no limitation of total memory at 2 GB, but I'am
not sure if I could have gone above 3 GB...
IDL>help,!version,/st
ARCH STRING 'x86'
OS STRING 'linux'
OS_FAMILY STRING 'unix'
OS_NAME STRING 'linux'
RELEASE STRING '6.0'
BUILD_DATE STRING 'Jun 27 2003'
MEMORY_BITS INT 32
FILE_OFFSET_BITS
INT 64
> One option I have is installing linux on this computer and dual booting,
> and perhaps I could do
> some tests on some of the other linux boxes around.
>
> Cheers,
> bob stockwell
>
>
--
____________________________________________________________ ________
Paolo Grigis
ETHZ - Institute of Astronomy email: pgrigis@astro.phys.ethz.ch
Scheuchzerstrasse 7
ETH Zentrum phone: ++41 1 632 42 20
8092 Zurich fax : ++41 1 632 12 05
Switzerland http://www.astro.phys.ethz.ch/
____________________________________________________________ ________
|
|
|
|
|
|
| Re: memory issues redux [message #41581 is a reply to message #41580] |
Mon, 15 November 2004 23:33 |
marc schellens[1]
Messages: 183
Registered: January 2000
|
Senior Member |
|
|
> 3) how are idl memory limitations under linux? Is it limited to 2gb, or
> could
> one access more memory than that? I am interested in both (maximum array
> size, and total memory available), if I could have 2 arrays of 1.3gb each,
> that
> would be awesome!
> One option I have is installing linux on this computer and dual booting,
> and perhaps I could do
> some tests on some of the other linux boxes around.
>
> Cheers,
> bob stockwell
A quick check here revealed, that IDL 6.0 und linux
(older Redhat, 2.4 kernel) IS limited to 2GB,
ie.
a=bytarr(1024,1024,2010)
works
a=bytarr(1024,1024,2020)
doesn't
And:
IDL> a=bytarr(1024,1024,1300)
IDL> b=bytarr(1024,1024,1300)
% Unable to allocate memory: to make array.
Cannot allocate memory
% Execution halted at: $MAIN$
HDH,
marc
|
|
|
|
| Re: memory issues redux - yay linux [message #41737 is a reply to message #41566] |
Thu, 18 November 2004 15:31 |
R.G.Stockwell
Messages: 163
Registered: October 2004
|
Senior Member |
|
|
Thanks to the groups for all the replies and information.
I ended up putting linux on in a dual boot configuration,
loading fedora core 3 with 4gb/4gb split, and check out these
results:
IDL> help
% At $MAIN$
A BYTE = Array[1024, 1024, 2047]
B BYTE = Array[1024, 1024, 1700]
C BYTE = Array[1024, 1024, 100]
D BYTE = Array[1024, 1024, 50]
IDL> help,/mem
heap memory used: 4086996992, max: 4086997008, gets: 859, frees:
459
WOW! Much nicer than winxp. Check out a, damn near a full 2gb.
and then next largest size I could get was ~1.7gb. Both blowing away the
best i could get in windows. I am off to the races.
Again, thanks for all the responses.
Cheers,
bob
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
