| Re: Variable assignment [message #35900] |
Tue, 22 July 2003 14:50 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
David Fanning <david@dfanning.com> writes:
> Craig Markwardt writes:
>
>> I routinely make index arrays that have millions, and tens of millions
>> of elements.
>
> What *is* it you guys do for a living!?
>
> I once made an index array that had about 100 points
> in it. But that's about as big as mine get. :-(
It's the difference between the data rich and the data poor, baby! :-)
Craig
P.S. To give a concrete example I've recently been looking at an 8
year mission timeline with 1 minute samples, or a total of 4 million
samples. I use WHERE to do data filtering.
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Variable assignment [message #35903 is a reply to message #35900] |
Tue, 22 July 2003 12:20  |
Haje Korth
Messages: 651
Registered: May 1997
|
Senior Member |
|
|
grrrrr, I applied the indices to the wrong level in the structure. I would
have expected an "out of range" error instead of "too many elements". I just
misinterpreted the message.
Thanks for helping,
Haje
--
"Craig Markwardt" <craigmnet@cow.physics.wisc.edu> wrote in message
news:onk7aawjw2.fsf@cow.physics.wisc.edu...
>
> "Haje Korth" <haje.korth@jhuapl.edu> writes:
>> I am trying to assign a variable with data=data(index), where index is
an
>> array with 300000 elements. IDL answers "Too many elements". Is there
away
>> around this? Basically, I need to kill some lines in data that contain
dummy
>> values. This the data files are HDF, I cannot eliminate these data when
the
>> file is read with hdf_read.
>
> I routinely make index arrays that have millions, and tens of millions
> of elements. [ with IDL 5.2 at that! ] This should not be a problem
> for you. Are you sure you have enough memory? You will probably need
> a fair amount of working space (perhaps 2x your biggest array) for IDL
> to function properly.
>
> Craig
>
> --
> ------------------------------------------------------------ --------------
> Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
> Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
> ------------------------------------------------------------ --------------
|
|
|
|
| Re: Variable assignment [message #35904 is a reply to message #35903] |
Tue, 22 July 2003 12:15 |
Haje Korth
Messages: 651
Registered: May 1997
|
Senior Member |
|
|
Craig,
I am pretty sure that I have enough RAM. I have almost 800 MB RAM and the
data I read in is less than 100 MB. If you say that you do not have a
problem, let me take a look again.
Thanks,
Haje
--
Dr. Haje Korth
Space Physics Group
The Johns Hopkins University
Applied Physics Laboratory
MS MP3-E128
11100 Johns Hopkins Road
Laurel, MD 20723-6099
USA
Phone: 240-228-4033 (Washington), 443-778-4033 (Baltimore)
Fax: 240-228-0386 (Washington), 443-778-0386 (Baltimore)
e-mail: haje.korth@jhuapl.edu
"Craig Markwardt" <craigmnet@cow.physics.wisc.edu> wrote in message
news:onk7aawjw2.fsf@cow.physics.wisc.edu...
>
> "Haje Korth" <haje.korth@jhuapl.edu> writes:
>> I am trying to assign a variable with data=data(index), where index is
an
>> array with 300000 elements. IDL answers "Too many elements". Is there
away
>> around this? Basically, I need to kill some lines in data that contain
dummy
>> values. This the data files are HDF, I cannot eliminate these data when
the
>> file is read with hdf_read.
>
> I routinely make index arrays that have millions, and tens of millions
> of elements. [ with IDL 5.2 at that! ] This should not be a problem
> for you. Are you sure you have enough memory? You will probably need
> a fair amount of working space (perhaps 2x your biggest array) for IDL
> to function properly.
>
> Craig
>
> --
> ------------------------------------------------------------ --------------
> Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
> Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
> ------------------------------------------------------------ --------------
|
|
|
|
| Re: Variable assignment [message #35905 is a reply to message #35904] |
Tue, 22 July 2003 10:37 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Craig Markwardt writes:
> I routinely make index arrays that have millions, and tens of millions
> of elements.
What *is* it you guys do for a living!?
I once made an index array that had about 100 points
in it. But that's about as big as mine get. :-(
Cheers,
David
--
David W. Fanning, Ph.D.
Fanning Software Consulting, Inc.
Phone: 970-221-0438, E-mail: david@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
| Re: Variable assignment [message #35906 is a reply to message #35905] |
Tue, 22 July 2003 10:28 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
"Haje Korth" <haje.korth@jhuapl.edu> writes:
> I am trying to assign a variable with data=data(index), where index is an
> array with 300000 elements. IDL answers "Too many elements". Is there away
> around this? Basically, I need to kill some lines in data that contain dummy
> values. This the data files are HDF, I cannot eliminate these data when the
> file is read with hdf_read.
I routinely make index arrays that have millions, and tens of millions
of elements. [ with IDL 5.2 at that! ] This should not be a problem
for you. Are you sure you have enough memory? You will probably need
a fair amount of working space (perhaps 2x your biggest array) for IDL
to function properly.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Variable assignment [message #35913 is a reply to message #35906] |
Tue, 22 July 2003 08:41 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Haje Korth writes:
> I am trying to assign a variable with data=data(index), where index is an
> array with 300000 elements. IDL answers "Too many elements". Is there away
> around this? Basically, I need to kill some lines in data that contain dummy
> values. This the data files are HDF, I cannot eliminate these data when the
> file is read with hdf_read.
Uh, at the risk of sounding impertinent, why don't
you just take smaller bites? :-)
Something like this:
indices = Where( array EQ BAD, count)
IF count GT 250000L THEN BEGIN ; Or whatever the number is
half = count/2
array[indices[0:half]] = -999
array[indices[half+1:*]] = -999
ENDIF ELSE array[indices] = -999
Cheers,
David
--
David W. Fanning, Ph.D.
Fanning Software Consulting, Inc.
Phone: 970-221-0438, E-mail: david@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
