| Re: Reading from process pipes in IDL? [message #21782] |
Thu, 21 September 2000 00:00 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
landsman@mpb.gsfc.nasa.gov writes:
> In V5.3, the /COMPRESS keyword was introduced to OPENR, which allows you
> to directly access gzip'ed files in an OS-independent way. I've been
> quite happy with this feature -- you can read compressed files almost
> completely transparently. One thing to watch out for, is that
> although POINT_LUN works on compressed files, it is much slower than on
> regular files, since IDL always has to reset to the beginning on each
> POINT_LUN call for compresed files.
I agree that this IDL v5.3 feature sounds pretty cool. I wonder when
we'll get it here... The seek slowdown you mentioned does not occur
in my seekable pipes library. The seek is lazy, which means that no
data is uncompressed until it actually needs to be read. You pay the
price however, by needing a temporary storage area to hold the
uncompressed data.
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Reading from process pipes in IDL? [message #21785 is a reply to message #21782] |
Thu, 21 September 2000 00:00  |
landsman
Messages: 93
Registered: August 1991
|
Member |
|
|
In article <39C9CBD4.D6CC859A@mpipsykl.mpg.de>,
Benno Puetz <puetz@mpipsykl.mpg.de> wrote:
> I have been trying to read image data from compressed (*.gz) files.
>
> While this is possible with
>
> SPAWN, "gunzip "+file+".gz"
>
> OPENR, l, file, /GETLUN
>
> READU, ....
>
> FREE_LUN, l
>
> I would prefer a way similar to PERL's
>
> open(FILE, "gzip -cd $file |");
>
> ...
>
> since it would not have to decompress/compress to disk.
>
> Is this at all possible with IDL?
In V5.3, the /COMPRESS keyword was introduced to OPENR, which allows you
to directly access gzip'ed files in an OS-independent way. I've been
quite happy with this feature -- you can read compressed files almost
completely transparently. One thing to watch out for, is that
although POINT_LUN works on compressed files, it is much slower than on
regular files, since IDL always has to reset to the beginning on each
POINT_LUN call for compresed files.
Wayne Landsman landsman@mpb.gsfc.nasa.gov
Sent via Deja.com http://www.deja.com/
Before you buy.
|
|
|
|
| Re: Reading from process pipes in IDL? [message #21790 is a reply to message #21782] |
Thu, 21 September 2000 00:00 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Benno Puetz <puetz@mpipsykl.mpg.de> writes:
> I have been trying to read image data from compressed (*.gz) files.
>
> While this is possible with
>
> SPAWN, "gunzip "+file+".gz"
>
> OPENR, l, file, /GETLUN
>
> READU, ....
>
> FREE_LUN, l
>
> I would prefer a way similar to PERL's
>
> open(FILE, "gzip -cd $file |");
>
> ...
>
> since it would not have to decompress/compress to disk.
>
> Is this at all possible with IDL?
If you are running on a Unix-type operating system, then you can give
my seekable pipes library a try. You can open a gzip'ed or
compress'ed file and they will automatically be unzipped on the fly as
you access them. You can also open any piped stream, not just gzipped
files.
In addition you can seek to anywhere in the (uncompressed) file or
pipe. The only restriction is that you must have enough temporary
disk space to hold the unzipped file.
Here it is, listed under Input/Output:
http://cow.physics.wisc.edu/~craigm/idl/idl.html
Craig
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: craigmnet@cow.physics.wisc.edu
Astrophysics, IDL, Finance, Derivatives | Remove "net" for better response
------------------------------------------------------------ --------------
|
|
|
|
| Re: Reading from process pipes in IDL? [message #21791 is a reply to message #21782] |
Thu, 21 September 2000 00:00 |
gfirem1
Messages: 1
Registered: September 2000
|
Junior Member |
|
|
Benno Puetz (puetz@mpipsykl.mpg.de) wrote:
: I have been trying to read image data from compressed (*.gz) files.
: I would prefer a way similar to PERL's
: open(FILE, "gzip -cd $file |");
Benno -
If you know the array size in advance, you can use the -c flag to gunzip
to stdout and capture the output in a unit pipe:
IDL> imagedata=intarr(1325,800)
IDL> spawn, 'gunzip -c testdata.dat.Z', unit=dataunit
IDL> readu,dataunit,imagedata
IDL> free_lun,dataunit
IDL> help,imagedata
IMAGEDATA INT = Array[1325, 800]
--
__.
/ | , , , , , ,_ Gwyn Fireman
(__/|/(_(_)(_/|/| |_/ gfirem1@alumni.umbc.edu
/| /|
(_/ (_/
|
|
|
|
| Re: Reading from process pipes in IDL? [message #21795 is a reply to message #21782] |
Thu, 21 September 2000 00:00 |
George N. White III
Messages: 56
Registered: September 2000
|
Member |
|
|
On Thu, 21 Sep 2000, Benno Puetz wrote:
> I have been trying to read image data from compressed (*.gz) files.
>
> While this is possible with
>
> SPAWN, "gunzip "+file+".gz"
>
> OPENR, l, file, /GETLUN
>
> READU, ....
>
> FREE_LUN, l
>
> I would prefer a way similar to PERL's
>
> open(FILE, "gzip -cd $file |");
>
> ...
>
> since it would not have to decompress/compress to disk.
>
> Is this at all possible with IDL?
>
> --
> Benno Puetz
> Kernspintomographie
> Max-Planck-Institut f. Psychiatrie Tel.: +49-89-30622-413
> Kraepelinstr. 10 Fax : +49-89-30622-520
> 80804 Muenchen, Germany
If your system (OS/2, unix, ??) supports named pipes, you can use them for
this sort of problem, although I have never tried them with IDL. Use
"mknod pipename p" to create a named pipe, then start a background process
to stuff data into the pipe, (e.g., "gzcat file.gz > pipename & "). In
your program, "pipename" is read as if it was an ordinary file, except
that you can't do things like Fortran backspace or rewind, or memory
mapped I/O.
--
George N. White III <gnw3@acm.org> Bedford Institute of Oceanography
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
