| Re: Large File Plotting [message #21887] |
Thu, 28 September 2000 13:52 |
Mark Hadfield
Messages: 783
Registered: May 1995
|
Senior Member |
|
|
"Mark Przeslawski" <MarkPrzeslawski@ncodeinc.com> wrote in message
news:st78offfa3gm3b@corp.supernews.com...
> Does anyone have an example of plotting
> very large time history data files. I tried
> creating a 1 million point file and it plotted
> okay. I tried a 10 million point file with
> various plotting routines and it hung my
> Windows 2000 machine (PIII, 700MHz,
> 256Mb RAM). I tried plot, live_plot,
> and some of the coyote plotting
> routines to no avail.
That's interesting. On my machine (not too different from yours: NT 4.0,
PII, 400 MHZ, 256 MB RAM) I find that if I can create the array I can plot
it:
IDL> plot, findgen(1000000)
IDL> plot, findgen(10000000)
IDL> plot, findgen(20000000)
IDL> plot, findgen(50000000)
% Unable to allocate memory: to make array.
% Execution halted at: $MAIN$
Of course, speed is an issue: about 3 s per million floating-point values on
my machine.
> ... What is needed is
> something that can detect file size and
> only display to the pixel resolution of the
> screen.
That's pretty much what the Direct Graphics plotting commands do. They just
write pixels to the screen as they go; they don't keep a copy of the data or
the plotting primitives. This is why IDL Direct Graphics is so good for
plotting really large datasets, and why it is so frustratingly limited in
some ways.
When plotting such large datasets you should avoid Object Graphics
(including LIVE_PLOT) and also Direct Graphics wrapper systems (like David's
XWINDOW) that copy the data. You will also need to be careful about
processing the data to avoid redundant copies, e.g. this generates two
10000000-element arrays
a = findgen(10000000)
b = a
but this generates only one
a = findgen(10000000)
b = temporary(a)
People other than me are more competent to advise you on handling large
datasets.
---
Mark Hadfield
m.hadfield@niwa.cri.nz http://katipo.niwa.cri.nz/~hadfield/
National Institute for Water and Atmospheric Research
PO Box 14-901, Wellington, New Zealand
Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
|
|
|
|
| Re: Large File Plotting [message #21888 is a reply to message #21887] |
Thu, 28 September 2000 00:00  |
davidf
Messages: 2866
Registered: September 1996
|
Senior Member |
|
|
Pavel Romashkin (promashkin@cmdl.noaa.gov) writes:
>> What is needed is
>> something that can detect file size and
>> only display to the pixel resolution of the
>> screen.
>
> I think this is exactly what you need. At 1024 x 768, a monitor is
> capable of producing not better than 85 points per inch. Resample your
> large array to the desired resolution and plot it. PLOT will do this
> anyway (on a visual level), but a lot less efficiently.
>
> If the axis you plot against is 5" long, then the maximum number of
> points you'll be able to see is < 500. Use CONGRID for plotting:
>
> plot, congrid(a, 500)
>
> works very fast.
Another suggestion along this line is to use the NSUM
keyword on the PLOT command. This indicates how many
adjacent points IDL is to average in order to reduce
the number of plotted points.
If you are in to widget programming, I think you
should also be able to write something along the
lines of ZPLOT on my web page that allows you to
"zoom" into your data at different "resolutions".
Some combination of resampling the actual data
points and restricting the independent data range
should work. I don't think this would be too hard
to write.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting
Phone: 970-221-0438 E-Mail: davidf@dfanning.com
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Toll-Free IDL Book Orders: 1-888-461-0155
|
|
|
|
| Re: Large File Plotting [message #21889 is a reply to message #21888] |
Thu, 28 September 2000 00:00 |
promashkin
Messages: 169
Registered: December 1999
|
Senior Member |
|
|
> What is needed is
> something that can detect file size and
> only display to the pixel resolution of the
> screen.
I think this is exactly what you need. At 1024 x 768, a monitor is
capable of producing not better than 85 points per inch. Resample your
large array to the desired resolution and plot it. PLOT will do this
anyway (on a visual level), but a lot less efficiently.
If the axis you plot against is 5" long, then the maximum number of
points you'll be able to see is < 500. Use CONGRID for plotting:
plot, congrid(a, 500)
works very fast.
Cheers,
Pavel
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
