| Re: IDL memory limitation? [message #45472] |
Thu, 15 September 2005 15:12  |
Karl Schultz
Messages: 341
Registered: October 1999
|
Senior Member |
|
|
On Thu, 15 Sep 2005 15:54:52 -0400, IDLmastertobe wrote:
> Thank you for answering my question. What I am confused now is that my
> file size is only 40-50MB instead of the 2-3GB mentioned. Why would I
> encounter memory problem? Is that related to I have multiple of these
> 40-50 MB files in my directory? I thought IDL is Direct Access and only
> load file when it is called. Thanks for your help.
It *can* be a pretty long path from the raw data to a visualization. In
other words, a number of operations are often applied to incoming data to
get it into the final visualization form. Perhaps you can tell us more
about what you are doing to the data to visualize it.
You said it was "3D" data. So, for example, you might be reading in the
data as a volume and then computing an isosurface. Depending on the
distribution of samples in the volume and the isovalue, you might end up
generating polygonal meshes containing many millions of triangles.
Karl
|
|
|
|
| Re: IDL memory limitation? [message #45473 is a reply to message #45472] |
Thu, 15 September 2005 14:29  |
Mark Hadfield
Messages: 783
Registered: May 1995
|
Senior Member |
|
|
IDLmastertobe wrote:
> Thanks Mike, I am doing it on a windows machine. Is there any way that I
> can check my memory usage for my program?a
>
There is an IDL procedure called memtest (or mem_test) that will report
on the areas of *contiguous* memory available to IDL. You can get it here:
http://www.rsinc.com/services/techtip.asp?ttid=3441
You might might want to look here as well:
http://www.rsinc.com/services/techtip.asp?ttid=3512
On my system (Windows 2000 Service Pack 4, 1 GiB RAM, a few GiB of swap
space), when IDL is freshly started, memtest reports the following
Memory block # 1: 1033 Mb (total: 1033 Mb)
Memory block # 2: 388 Mb (total: 1421 Mb)
Memory block # 3: 203 Mb (total: 1624 Mb)
Memory block # 4: 61 Mb (total: 1685 Mb)
Memory block # 5: 58 Mb (total: 1743 Mb)
Memory block # 6: 48 Mb (total: 1791 Mb)
Memory block # 7: 33 Mb (total: 1824 Mb)
Memory block # 8: 21 Mb (total: 1845 Mb)
Memory block # 9: 20 Mb (total: 1865 Mb)
Memory block #10: 17 Mb (total: 1882 Mb)
(where the "Mb"s are supposed to be "MB"s, ie megabytes rather than
megabits).
*However* from time to time IDL gets into a state where "Memory block
#1" is reduced to only 350 MiB or so. A full restart does not fix
this--the only cure is to restart IDL. I haven't reported this problem
to RSI yet, as I don't know how to reproduce it. (I *suspect* it may
have something to do with the video driver, as it seems to occur when I
am trying to do heavy-duty object graphics--if this is true going back
to software rendering may fix it, but, like I said, I cannot reproduce
it reliably so it's hard to diagnose or solve it.)
Finally you say the data you are visualising total only 40-50 MiB. With
IDL it is easy to do things in such a way that the total memory
reequirement exceeds the size of your data many times, by inadvertently
making copies of your data and such like. (If it's any consolation, this
problem is *much* worse in Matlab.) Object Graphics tends to be pretty
heavy on memory. I suggest you experiment with smaller datasets, using
"help, /MEMORY" to establish the memory requirements.
--
Mark Hadfield "Kei puwaha te tai nei, Hoea tahi tatou"
m.hadfield@niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)
|
|
|
|
|
|
|
|
|
|
|
|
| Re: IDL memory limitation? [message #45485 is a reply to message #45477] |
Thu, 15 September 2005 08:26 |
R.G. Stockwell
Messages: 363
Registered: July 1999
|
Senior Member |
|
|
"IDLmastertobe" <shi_lee@hotmail.com> wrote in message
news:350f4523765a1e1dc7f21a03bc5e7f17@localhost.talkaboutpro gramming.com...
> Hi, I am trying to visualize some 3D data using IDL and I found when my
> data size grow big, the program runs out of memory and won't work.
> Indeed, the data size is not too big yet, it is about 50MB max while some
> of my other data could be on the level of few or more GigaBytes in the
> future. Does anyone have any idea if IDL has any certain limitations on
> memory allocation? Thank you.
>
Yes, there are "extreme" limitations on IDL memory allocation.
Under windows, it is pretty tough, as windows sprays dlls all through
your ram, making the largest contiguous piece pretty small (I could
only make a 700 mb array, and 2gb is the absolute max a process
can access under windows).
Your much better off in under *nix, under fedora core 4 I can
allocate almost all 4 gigs of ram, with the largest arrays being
just over a gig. (4 gig being the max under 32 bit OS)
I don't know the status of 64 bit idl, but I think it is available.
If you need huge memory allocations, definitely look at a
new 64 bit computer, 64 bit os, and 64 bit IDL .
Cheers,
bob
PS there was a thread titled memory issues redux
that discussed this.
(link to groups.google below)
http://tinyurl.com/a2r5q
|
|
|
|
|
|
| Re: IDL memory limitation? [message #45644 is a reply to message #45472] |
Fri, 16 September 2005 14:18 |
IDLmastertobe
Messages: 54
Registered: June 2004
|
Member |
|
|
Thank you everyone, here is my test result from memtest:
Memory block # 1: 569 Mb (total: 569 Mb)
Memory block # 2: 382 Mb (total: 951 Mb)
Memory block # 3: 196 Mb (total: 1147 Mb)
Memory block # 4: 183 Mb (total: 1330 Mb)
Memory block # 5: 128 Mb (total: 1458 Mb)
Memory block # 6: 88 Mb (total: 1546 Mb)
Memory block # 7: 63 Mb (total: 1609 Mb)
Memory block # 8: 59 Mb (total: 1668 Mb)
Memory block # 9: 40 Mb (total: 1708 Mb)
Memory block #10: 37 Mb (total: 1745 Mb)
before I was visualing 3-D data on the size of say
50x50x10, now im visualizing data on the size of 250x250x1 and it is
giving me error saying it can't allocate memory to create array. I check
the memory by using "help, /memory" and found the heap memory is used up.
I am taking in 3D data and visualizing it by using IDLgrContainer. I
created IDLgrModel and IDLgrAxis etc and put them together to visualize
them. I can rotate it or flip it any way I want. It is a real time
visualization. Does anyone know how I can overcome this memory problem?
I currently have 1GB of RAM installed and 2GB of Virtual Memory allocated.
Thanks.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
