| Strange memory problem [message #18337] |
Tue, 21 December 1999 00:00  |
rkj
Messages: 66
Registered: February 1996
|
Member |
|
|
When I do the following:
temp=bytarr(1000,1000,25)
temp(*)=10
my computer (a Sparc20 with 128MB of *available* ram) grinds
to a halt. Actually it starts using swap. Don't tell me
PV-Wave is making 5 or 6 copies of the array just to do this
simple process!
Has anyone else seen this curious behavior on PV-Wave? IDL? I have
verified it on a PC, and using different versions of PV-Wave(6.1,7.0).
I have played around with the z-value and 25 (i.e. 25MB) is roughly
the cutoff for going into swap for a 128MB system.
Kyle
|
|
|
|
| Re: Strange memory problem [message #18403 is a reply to message #18337] |
Thu, 23 December 1999 00:00  |
Terry Smith
Messages: 2
Registered: December 1999
|
Junior Member |
|
|
----------
In article <gdj84.46$O3.1036@uchinews>, rivers@cars3.uchicago.edu (Mark
Rivers) wrote:
> In article <83rfkn$826$1@nnrp1.deja.com>, Peter Mason
> <menakkis@my-deja.com> writes:
>> rkj@dukebar.crml.uab.edu (R. Kyle Justice) wrote:
>> <...>
>>> Actually I should have given my real problem rather than a
>>> simplified version of it. Acutally I have two big arrays of
>>> equal size and I am trying to copy one into the other:
>>>
>>> temp1(*)=temp2
>> <...>
>>
>> The best way I know of to do this sort of thing (in IDL) is to use
>> array-insertion starting offsets.
>> e.g., If you have 2-dimensional arrays and you want to copy one on top
>> of another, do: TEMP1(0,0)=TEMP2.
>> This is much, *much* more efficient than using '*'. It also converts
>> what it copies to the datatype of TEMP1 if necessary.
>> This issue goes back a long way, so I'd expect the "solution" to work
>> on PV-Wave as well.
>
> This is definitely the best way to do this. Note that you don't need both
> subscripts. You can just say:
> a = bytarr(100,100,25)+10b ; Create array of all 10's
> b = bytarr(100,100,25) ; Create a zero filled array
> b(0) = a ; This copies a into b
>
>
> Mark Rivers
This is faster than the straightforward:
b=a
or
b=temporary(a)
???
JD
>
|
|
|
|
| Re: Strange memory problem [message #18411 is a reply to message #18337] |
Thu, 23 December 1999 00:00 |
rivers
Messages: 228
Registered: March 1991
|
Senior Member |
|
|
In article <83rfkn$826$1@nnrp1.deja.com>, Peter Mason <menakkis@my-deja.com> writes:
> rkj@dukebar.crml.uab.edu (R. Kyle Justice) wrote:
> <...>
>> Actually I should have given my real problem rather than a
>> simplified version of it. Acutally I have two big arrays of
>> equal size and I am trying to copy one into the other:
>>
>> temp1(*)=temp2
> <...>
>
> The best way I know of to do this sort of thing (in IDL) is to use
> array-insertion starting offsets.
> e.g., If you have 2-dimensional arrays and you want to copy one on top
> of another, do: TEMP1(0,0)=TEMP2.
> This is much, *much* more efficient than using '*'. It also converts
> what it copies to the datatype of TEMP1 if necessary.
> This issue goes back a long way, so I'd expect the "solution" to work
> on PV-Wave as well.
This is definitely the best way to do this. Note that you don't need both
subscripts. You can just say:
a = bytarr(100,100,25)+10b ; Create array of all 10's
b = bytarr(100,100,25) ; Create a zero filled array
b(0) = a ; This copies a into b
Mark Rivers
|
|
|
|
| Re: Strange memory problem [message #18414 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
jeyadev
Messages: 78
Registered: February 1995
|
Member |
|
|
In article <38602B16.5EDA39B@resource-eng.com>,
Mark D. Williams <markw-xxxnospamxxx@resource-eng.com> wrote:
> "R. Kyle Justice" wrote:
>>
>> When I do the following:
>>
>> temp=bytarr(1000,1000,25)
>> temp(*)=10
>>
>> my computer (a Sparc20 with 128MB of *available* ram) grinds
>> to a halt. Actually it starts using swap. Don't tell me
>> PV-Wave is making 5 or 6 copies of the array just to do this
>> simple process!
>>
>> Has anyone else seen this curious behavior on PV-Wave? IDL? I have
>> verified it on a PC, and using different versions of PV-Wave(6.1,7.0).
>>
>> I have played around with the z-value and 25 (i.e. 25MB) is roughly
>> the cutoff for going into swap for a 128MB system.
Just tried it on my Ultra10 with 1024 MB of RAM -- not even running
Netscape, so there must be enough memory! Same problem -- swapping
does occur and the it takes about 15 seconds to perform the operation.
I am running Solaris 2.6 and PV-WAVE v6.01.
--
Surendar Jeyadev jeyadev@wrc.xerox.com
|
|
|
|
| Re: Strange memory problem [message #18417 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
Mark D. Williams
Messages: 15
Registered: November 1999
|
Junior Member |
|
|
"R. Kyle Justice" wrote:
> Actually I should have given my real problem rather than a
> simplified version of it. Actually I have two big arrays of
> equal size and I am trying to copy one into the other:
>
> temp1(*)=temp2
>
> I don't want memory fragmentation, that is why I do the process
> "in place." I was told by one person that this left hand operation
> generates an array of indices and this is where the memory problem
> is coming from. He suggested the IDL (I am assuming) function
> REPLICATE_INPLACE. Well, I don't have that on PV-Wave(or do I?)
> and it really is not helpful for my actually problem (only the
> simplified one). So it appears that I am stuck with fragmentation
> or disk swapping. Choose your poison.
It would appear so, although by my view, in this case fragmentation
is the lesser of the two evils.
> : By assigning 10 instead of 10B, that is what you're ending up with.
>
> I thought including the * operator on the left side would
> preserve the data type.
You are, of course, correct. It is also true as another poster
pointed out, that for the simplified example, using the /NoZero
keyword isn't really a fair test since you end up with an actual
value being "added" to uninitialized memory space.
Regards,
Mark Williams
Resource Engineering, Inc.
|
|
|
|
| Re: Strange memory problem [message #18418 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
m218003
Messages: 56
Registered: August 1999
|
Member |
|
|
In article <83qsal$r3k$1@sonofmaze.dpo.uab.edu>,
rkj@dukebar.crml.uab.edu (R. Kyle Justice) writes:
> Mark D. Williams (markw-xxxnospamxxx@resource-eng.com) wrote:
> : "R. Kyle Justice" wrote:
>
> Actually I should have given my real problem rather than a
> simplified version of it. Acutally I have two big arrays of
> equal size and I am trying to copy one into the other:
>
> temp1(*)=temp2
>
>
On a fresh system (Sun ULTRA5), I got the following responses:
IDL> t0=systime(1) & temp5 = BYTARR(1000,1000,25) + 10B & t1=systime(1)
IDL> print,t1-t0
0.65915895
IDL> t0=systime(1) & temp = temp5 & t1=systime(1) & print,t1-t0
0.35241199
IDL> t0=systime(1) & temp = byte(temp5) & t1=systime(1) & print,t1-t0
0.15835595
The next attempt, to do
IDL> t0=systime(1) & temp[*] = temp5 & t1=systime(1) & print,t1-t0
led to happy swapping ... And when I start another new IDL session,
the sequence
IDL> t0=systime(1) & temp5 = BYTARR(1000,1000,25) + 10B & t1=systime(1) & print,t1-t0
0.66182601
IDL> t0=systime(1) & temp = BYTARR(1000,1000,25,/NOZERO) & t1=systime(1) & print,t1-t0
0.00050103664
IDL> t0=systime(1) & temp[*] = temp5 & t1=systime(1) & print,t1-t0
again leads to a system breakdown. Oh! after idling for 93 seconds, the
command prompt reappeared.
Cheers,
Martin
--
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
[[ Dr. Martin Schultz Max-Planck-Institut fuer Meteorologie [[
[[ Bundesstr. 55, 20146 Hamburg [[
[[ phone: +49 40 41173-308 [[
[[ fax: +49 40 41173-298 [[
[[ martin.schultz@dkrz.de [[
[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [[[[[[[
|
|
|
|
| Re: Strange memory problem [message #18419 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
Peter Mason
Messages: 145
Registered: June 1996
|
Senior Member |
|
|
rkj@dukebar.crml.uab.edu (R. Kyle Justice) wrote:
<...>
> Actually I should have given my real problem rather than a
> simplified version of it. Acutally I have two big arrays of
> equal size and I am trying to copy one into the other:
>
> temp1(*)=temp2
<...>
The best way I know of to do this sort of thing (in IDL) is to use
array-insertion starting offsets.
e.g., If you have 2-dimensional arrays and you want to copy one on top
of another, do: TEMP1(0,0)=TEMP2.
This is much, *much* more efficient than using '*'. It also converts
what it copies to the datatype of TEMP1 if necessary.
This issue goes back a long way, so I'd expect the "solution" to work
on PV-Wave as well.
Cheers
Peter Mason
Sent via Deja.com http://www.deja.com/
Before you buy.
|
|
|
|
| Re: Strange memory problem [message #18420 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
thompson
Messages: 584
Registered: August 1991
|
Senior Member |
|
|
"Julie Greenwood" <julieg@NOSPAMoceanweather.com> writes:
> Liam Gumley <Liam.Gumley@ssec.wisc.edu> wrote in message
> news:3860E331.C4E36EA4@ssec.wisc.edu...
>> "Mark D. Williams" wrote:
>>> FWIW, if you want to save time and memory, a faster way to do the above
>>> is as follows:
>>>
>>> WAVE> temp = BYTARR(1000,1000,25, /NoZero) + 10B
>>
>> I get the fastest response (in IDL) with
>>
>> temp = replicate(10B, 1000, 1000, 25)
>>
>> Cheers,
>> Liam.
> With IDL 5.3 on WinNT I got:
> temp = BYTARR(1000,1000,25, /NoZero) + 10B
> 1.8230000 Seconds
> temp = replicate(10B, 1000, 1000, 25)
> 6.3290000 Seconds
> I got tired of waiting (more than two minutes) for
> temp=bytarr(1000,1000,25) ; 0.351 seconds for this much
> temp(*)=10
I suggest redoing the test with
temp = BYTARR(1000,1000,25) + 10B
with the /NoZero keyword you get a much faster response, but nonsense values.
It's not really a fair test.
On my AlphaStation 500/333, I got the following times:
temp = BYTARR(1000,1000,25) + 10B
1.8984600 Seconds
temp = replicate(10B, 1000, 1000, 25)
1.2076780 Seconds
temp=bytarr(1000,1000,25)
temp(*)=10
9.2365350 Seconds
temp=bytarr(1000,1000,25)
temp(*)=10b
8.4264030 Seconds
Bill Thompson
|
|
|
|
| Re: Strange memory problem [message #18422 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
Julie Greenwood
Messages: 6
Registered: December 1999
|
Junior Member |
|
|
Liam Gumley <Liam.Gumley@ssec.wisc.edu> wrote in message
news:3860E331.C4E36EA4@ssec.wisc.edu...
> "Mark D. Williams" wrote:
>> FWIW, if you want to save time and memory, a faster way to do the above
>> is as follows:
>>
>> WAVE> temp = BYTARR(1000,1000,25, /NoZero) + 10B
>
> I get the fastest response (in IDL) with
>
> temp = replicate(10B, 1000, 1000, 25)
>
> Cheers,
> Liam.
OK, After I figured out to clear the memory between calls, I got the
following numbers (before clearing the temp array, the numbers just kept
growing...):
0.33000004 Seconds (temp=bytarr(1000,1000,25))
1.5920000 Seconds (replicate_inplace, temp, 10B)
2.0829999 Seconds (temp = BYTARR(1000,1000,25) + 10B)
1.9130000 Seconds (temp = BYTARR(1000,1000,25, /NoZero) + 10B)
1.5619999 Seconds (temp = replicate(10B, 1000, 1000, 25))
pro test_array_init
T = SYSTIME(1)
temp=bytarr(1000,1000,25)
PRINT, SYSTIME(1) - T, ' Seconds (temp=bytarr(1000,1000,25))'
temp = 0
; the following takes a *very* long time
;temp(*)=10B
;PRINT, SYSTIME(1) - T, ' Seconds'
T = SYSTIME(1)
temp=bytarr(1000,1000,25)
replicate_inplace, temp, 10B
PRINT, SYSTIME(1) - T, ' Seconds (replicate_inplace, temp, 10B)'
temp = 0
T = SYSTIME(1)
temp = BYTARR(1000,1000,25) + 10B
PRINT, SYSTIME(1) - T, ' Seconds (temp = BYTARR(1000,1000,25) + 10B)'
temp = 0
T = SYSTIME(1)
temp = BYTARR(1000,1000,25, /NoZero) + 10B
PRINT, SYSTIME(1) - T, ' Seconds (temp = BYTARR(1000,1000,25, /NoZero) +
10B)'
temp = 0
T = SYSTIME(1)
temp = replicate(10B, 1000, 1000, 25)
PRINT, SYSTIME(1) - T, ' Seconds (temp = replicate(10B, 1000, 1000, 25))'
temp = 0
end
Julie
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Juliet G. Greenwood
Senior Programmer
Oceanweather Inc.
JulieG@Oceanweather.com
http://www.oceanweather.com/
http://www.thehungersite.com/
The Hunger Site - Donate Food for Free to
Feed Hungry People in the World.
The Site's Sponsors Make the Donations.
|
|
|
|
| Re: Strange memory problem [message #18423 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
Alex Schuster
Messages: 124
Registered: February 1997
|
Senior Member |
|
|
"R. Kyle Justice" wrote:
> When I do the following:
>
> temp=bytarr(1000,1000,25)
> temp(*)=10
>
> my computer (a Sparc20 with 128MB of *available* ram) grinds
> to a halt. Actually it starts using swap. Don't tell me
> PV-Wave is making 5 or 6 copies of the array just to do this
> simple process!
>
> Has anyone else seen this curious behavior on PV-Wave? IDL? I have
> verified it on a PC, and using different versions of PV-Wave(6.1,7.0).
Yes, I encountered the same problem. It seems that, when using the (*)
notation, IDL (and probably also PV-WAVE) replaces the * by an
appropiate index list, like here:
temp = bytarr(1000,1000,25)
index = lindgen(25000000L)
temp(index) = 10B
This index list takes 100 MB of memory! You can check this by the
command HELP, /MEMORY.
As others have pointed out, it is much faster when you use something
like this:
temp = bytarr(1000,1000,25)+10B
(Don't forget the B, because you would get an INTARR instead.)
This is much faster, and you need only 25 MB of RAM, as one would
expect.
Alex
--
Alex Schuster Wonko@weird.cologne.de PGP Key available
alex@pet.mpin-koeln.mpg.de
|
|
|
|
| Re: Strange memory problem [message #18424 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
thompson
Messages: 584
Registered: August 1991
|
Senior Member |
|
|
"Mark D. Williams" <markw-xxxnospamxxx@resource-eng.com> writes:
> "R. Kyle Justice" wrote:
>>
>> When I do the following:
>>
>> temp=bytarr(1000,1000,25)
>> temp(*)=10
>>
>> my computer (a Sparc20 with 128MB of *available* ram) grinds
>> to a halt. Actually it starts using swap. Don't tell me
>> PV-Wave is making 5 or 6 copies of the array just to do this
>> simple process!
>>
>> Has anyone else seen this curious behavior on PV-Wave? IDL? I have
>> verified it on a PC, and using different versions of PV-Wave(6.1,7.0).
>>
>> I have played around with the z-value and 25 (i.e. 25MB) is roughly
>> the cutoff for going into swap for a 128MB system.
>>
>> Kyle
> I don't have access to a Sparc, but I tried this on PV-WAVE 7.0 on
> both Windows NT 4.0 and RedHat Linux 6.0 on a system with 256 Mb of
> physical RAM and didn't experience any problem: i.e., no swapping,
> returned to the prompt within 3-4 seconds.
> FWIW, if you want to save time and memory, a faster way to do the above
> is as follows:
> WAVE> temp = BYTARR(1000,1000,25, /NoZero) + 10B
^^^^^^^
Should this /NOZERO be here? I don't know about PVWAVE, but in IDL, this
causes the array to be created without initializing the memory. You end up
with an array, but filled with nonsense values. I think what you meant was
WAVE> temp = BYTARR(1000,1000,25) + 10B
> Note, was it your intent to end up with temp being an INTARR? By
> assigning
> 10 instead of 10B, that is what you're ending up with.
Again, I don't know anything about PVWAVE, but in IDL the statement
>> temp(*)=10
does not change the type of the array temp. Since temp was created as a byte
array, the integer value 10 is converted to type byte before storing into the
array.
I'm not sure about what's happening internally when a command like temp(*)=10
is used, but it may be that a temporary index array is created to store the
positions of all the referenced points. It would have to be a long array, so
it would use up four times as much space as the byte array. That's just my
guess.
As stated before, the best way in IDL to do this would be
temp = replicate(10b, 1000,1000,25)
William Thompson
|
|
|
|
| Re: Strange memory problem [message #18425 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
Julie Greenwood
Messages: 6
Registered: December 1999
|
Junior Member |
|
|
Liam Gumley <Liam.Gumley@ssec.wisc.edu> wrote in message
news:3860E331.C4E36EA4@ssec.wisc.edu...
> "Mark D. Williams" wrote:
>> FWIW, if you want to save time and memory, a faster way to do the above
>> is as follows:
>>
>> WAVE> temp = BYTARR(1000,1000,25, /NoZero) + 10B
>
> I get the fastest response (in IDL) with
>
> temp = replicate(10B, 1000, 1000, 25)
>
> Cheers,
> Liam.
With IDL 5.3 on WinNT I got:
temp = BYTARR(1000,1000,25, /NoZero) + 10B
1.8230000 Seconds
temp = replicate(10B, 1000, 1000, 25)
6.3290000 Seconds
I got tired of waiting (more than two minutes) for
temp=bytarr(1000,1000,25) ; 0.351 seconds for this much
temp(*)=10
Julie
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Juliet G. Greenwood
Senior Programmer
Oceanweather Inc.
JulieG@Oceanweather.com
http://www.oceanweather.com/
http://www.thehungersite.com/
The Hunger Site - Donate Food for Free to
Feed Hungry People in the World.
The Site's Sponsors Make the Donations.
|
|
|
|
| Re: Strange memory problem [message #18426 is a reply to message #18337] |
Wed, 22 December 1999 00:00 |
rkj
Messages: 66
Registered: February 1996
|
Member |
|
|
Mark D. Williams (markw-xxxnospamxxx@resource-eng.com) wrote:
: "R. Kyle Justice" wrote:
: >
: > When I do the following:
: >
: > temp=bytarr(1000,1000,25)
: > temp(*)=10
: >
: > my computer (a Sparc20 with 128MB of *available* ram) grinds
: > to a halt. Actually it starts using swap. Don't tell me
: > PV-Wave is making 5 or 6 copies of the array just to do this
: > simple process!
: >
: > Has anyone else seen this curious behavior on PV-Wave? IDL? I have
: > verified it on a PC, and using different versions of PV-Wave(6.1,7.0).
: >
: > I have played around with the z-value and 25 (i.e. 25MB) is roughly
: > the cutoff for going into swap for a 128MB system.
: >
: > Kyle
: I don't have access to a Sparc, but I tried this on PV-WAVE 7.0 on
: both Windows NT 4.0 and RedHat Linux 6.0 on a system with 256 Mb of
: physical RAM and didn't experience any problem: i.e., no swapping,
: returned to the prompt within 3-4 seconds.
: FWIW, if you want to save time and memory, a faster way to do the above
: is as follows:
: WAVE> temp = BYTARR(1000,1000,25, /NoZero) + 10B
Actually I should have given my real problem rather than a
simplified version of it. Acutally I have two big arrays of
equal size and I am trying to copy one into the other:
temp1(*)=temp2
I don't want memory fragmentation, that is why I do the process
"in place." I was told by one person that this left hand operation
generates an array of indices and this is where the memory problem
is coming from. He suggested the IDL (I am assuming) function
REPLICATE_INPLACE. Well, I don't have that on PV-Wave(or do I?)
and it really is not helpful for my actually problem (only the
simplified one). So it appears that I am stuck with fragmentation
or disk swapping. Choose your poison.
: Note, was it your intent to end up with temp being an INTARR? By
: assigning
: 10 instead of 10B, that is what you're ending up with.
I thought including the * operator on the left side would
preserve the data type.
: Regards,
: Mark Williams
: Resource Engineering, Inc.
Thanks,
Kyle J.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
