| Re: Avoiding memory paging with large data-set [message #6272] |
Thu, 23 May 1996 00:00 |
sterner
Messages: 106
Registered: February 1991
|
Senior Member |
|
|
David Foster <foster@bial1.ucsd.edu> writes:
> I have a question for those familiar with UNIX systems stuff. We
> are using SPARC 10s and 20s under Solaris 2.3, and often have to
> work with very large data-sets. Sometimes the operations go pretty
> fast, and sometimes they go REALLY slow! Same machine, even the
> same data, but the time varies. I can tell that the machine is
> having to page the memory in and out continuously when things
> slow down, but I don't know why.
One thing to consider is in what order you access the data.
If you can access along the first dimension of an array it
may be quicker than along another dimension. For example,
access a large 2-d array in x instead of y if possible. I found
a case where I could speed things up significantly by doing a
transpose on a 2-d array before access the data. I could extract
rows then instead of columns and even with the added transpose
time it was faster. You will have to experiment with this.
. . .
I just tried an example for you:
a = lindgen(500,500)
t0=systime(1) & for i=0,499 do z=a(i,*) & print,systime(1)-t0
; 0.35461497 <- seconds.
t0=systime(1) & for i=0,499 do z=a(*,i) & print,systime(1)-t0
; 0.031723022 <- seconds.
That was on a loaded system (HP 7/35, two IDL jobs, plus a tar).
And I don't think the test run was paging.
Ray Sterner sterner@tesla.jhuapl.edu
The Johns Hopkins University North latitude 39.16 degrees.
Applied Physics Laboratory West longitude 76.90 degrees.
Laurel, MD 20723-6099
WWW Home page: http://fermi.jhuapl.edu/s1r/people/res/res.html
|
|
|
|
| Re: Avoiding memory paging with large data-set [message #6279 is a reply to message #6272] |
Wed, 22 May 1996 00:00  |
todd
Messages: 25
Registered: February 1994
|
Junior Member |
|
|
In article <4nvvgn$7jc@fu-berlin.de>, marq@fub46.zedat.fu-berlin.de (Christian Marquardt) writes:
|> Hello,
|>
|> David Foster (foster@bial1.ucsd.edu) wrote:
|> : Hello all.
|>
|> : I have a question for those familiar with UNIX systems stuff. We
|> : are using SPARC 10s and 20s under Solaris 2.3, and often have to
|> : work with very large data-sets. Sometimes the operations go pretty
|>
|> [stuff deleted]
|>
|> : but the intermittency is strange. Any help in how to analyze
|> : the system's "state", including memory allocated and for which
|> : processes, would be greatly appreciated.
|>
|> apart from ps, theres the vmstat command, showing (e.g) paging activity
|> or swapping activity, every few seconds or so...
|>
|> Hope this helps.
|>
|> Chris Marquardt (marq@strat01.met.fu-berlin.de)
|>
There also exists and OpenWindows interface called proctool.
It seems to be a GUI for ps, vmstat and others of that ilk.
It even works for multi-processors. It's a pretty slick
tool. Try an Archie search for it. If you can't find it, let
me know and I'll see if I can dig up an anonymous ftp address
for you.
--
Todd Ratcliff | (310)825-3118
UCLA Geodynamics Research Group | todd@artemis.ess.ucla.edu
Dept. of Earth & Space Sciences | http://artemis.ess.ucla.edu/~todd
|
|
|
|
| Re: Avoiding memory paging with large data-set [message #6280 is a reply to message #6279] |
Wed, 22 May 1996 00:00 |
marq
Messages: 12
Registered: February 1996
|
Junior Member |
|
|
Hello,
David Foster (foster@bial1.ucsd.edu) wrote:
: Hello all.
: I have a question for those familiar with UNIX systems stuff. We
: are using SPARC 10s and 20s under Solaris 2.3, and often have to
: work with very large data-sets. Sometimes the operations go pretty
[stuff deleted]
: but the intermittency is strange. Any help in how to analyze
: the system's "state", including memory allocated and for which
: processes, would be greatly appreciated.
apart from ps, theres the vmstat command, showing (e.g) paging activity
or swapping activity, every few seconds or so...
Hope this helps.
Chris Marquardt (marq@strat01.met.fu-berlin.de)
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
