comp.lang.idl-pvwave archive: archive » How to read a file in zip or gz format using IDL?

Home » Public Forums » archive » How to read a file in zip or gz format using IDL?

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

How to read a file in zip or gz format using IDL? [message #2507]

Wed, 03 August 1994 08:17

j.tan
Messages: 4
Registered: August 1994

Junior Member

idlusers-news-gateway:
id AA05653; Wed, 3 Aug 94 17:18:53 +0200
Wed, 03 Aug 94 16:18:25 BST
X-Sender: tan@rcru1.te.rl.ac.uk
Mime-Version: 1.0
X-Mailer: <Windows Eudora Version 2.0.2>

Greetings,

I am asking for help to read a file in zip or gz format using IDL. At the
moment, I have a huge data file in ASCII format about 10 Mbytes, it thus
takes a long time to read in all the data. I wonder it may take less time if
the data is compressed using either zip or gzip methods. But how can I read
such compressed file? Any advice over there is highly appreciated.

All the best,

--
Jun Tan
Rutherford Appleton Laboratory
j.tan@rutherford.ac.uk

Report message to a moderator

Re: How to read a file in zip or gz format using IDL? [message #2508 is a reply to message #2507]

Wed, 03 August 1994 14:03

tai
Messages: 14
Registered: June 1994

Junior Member

Tan, Jun (j.tan@rutherford.ac.uk) wrote:
: I am asking for help to read a file in zip or gz format using IDL. At the
: moment, I have a huge data file in ASCII format about 10 Mbytes, it thus
: takes a long time to read in all the data. I wonder it may take less time if
: the data is compressed using either zip or gzip methods. But how can I read
: such compressed file? Any advice over there is highly appreciated.

10 megs, yikes! I would recommend using restore (after an initial
save); i.e. load your ASCII data into memory, then do a

save, bigdata, filename = 'bigdata.sav'

Then, whenever you need to load it in, use

restore, filename = 'bigdata.sav'

I've found this to be faster than any other method.

Alan

Report message to a moderator

Re: How to read a file in zip or gz format using IDL? [message #2620 is a reply to message #2507]

Thu, 04 August 1994 08:34

8015
Messages: 52
Registered: November 1993

Member

In article <31ocemINNbqu@maz4.sma.ch>, Tan, Jun <j.tan@rutherford.ac.uk> wrote:
>
> I am asking for help to read a file in zip or gz format using IDL. At the
> moment, I have a huge data file in ASCII format about 10 Mbytes, it thus
> takes a long time to read in all the data. I wonder it may take less time if
> the data is compressed using either zip or gzip methods. But how can I read
> such compressed file? Any advice over there is highly appreciated.
>

I'll start by assuming we're referring to IDL in a unix environment.
You can use the spawn command to manager this. Create a command string
such as:

cmd = "uncompress -c comp_file" ; comp_file is the compressed file
spawn, cmd, uncomp_file ; uncomp_file will have the uncompressed data

The -c decompresses to stdout. Also the zcat command (linked to
uncompress) will do the same thing. This way you won't have to 'rm' the
file afterwards. I'm sure gzip has the same facility and could be used
just as easily.

Mike Schienle Hughes Santa Barbara Research Center
8015@sbsun0010.sbrc.hac.com 75 Coromar Drive, M/S B28/87
Voice: (805)562-7466 Fax: (805)562-7881 Goleta, CA 93117

Report message to a moderator

Re: How to read a file in zip or gz format usin [message #2640 is a reply to message #2507]

Fri, 05 August 1994 08:33

chase
Messages: 62
Registered: May 1993

Member

>>>> > "Thomas" == Thomas Oettli <oet@sma.ch> writes:

Thomas> No problem, I had a similiar problem some time ago, here is
Thomas> the data read procedure I used,

Thomas> FUNCTION GET_RC, f
Thomas> ; Return reference coordinates
Thomas> coords = { x_ref:fltarr(305,269), y_ref:fltarr(305,269) }
Thomas> spawn, ['gunzip','-c',f], UNIT=lun, /NOSHELL
Thomas> readu, lun, coords
Thomas> free_lun, lun return, coords
Thomas> END

I have tried this in the past but have experienced problems with the
IDL prematurely losing the pipe. This seems to happen when I try to
do other file I/O before I finish using pipe created by the spawn
command.

Has anyone else experienced difficulties with using UNIX pipes created
with spawn?

Chris
--
===============================
Bldg 24-E188
The Applied Physics Laboratory
The Johns Hopkins University
Laurel, MD 20723-6099
(301)953-6000 x8529
chris_chase@jhuapl.edu

Report message to a moderator

Re: How to read a file in zip or gz format using IDL? [message #2645 is a reply to message #2507]

Fri, 05 August 1994 05:33

j.tan
Messages: 4
Registered: August 1994

Junior Member

Hi, everyone,

I really appreciate all the advice I received since I posted my original
question, either through private e-mails or appearing in here. What a friendly
newsgroup!

However, the advice to use 'spawn' would not help me since I am using IDL for
Windows, in which the keyword of "unit" in spawn command is forbidden. I know
"spawn 'gzip -c .....' unit=lun" can be used in Unix as the manual tells me.
In MS-DOS, the "spawn" is limited to the output of line number, not as the
file unit in Unix.

Well, I will try to convert that ASCII file to binary format so IDL can read
quick, at least for the time being.

All the best,

Jun Tan

--
Jun Tan

Report message to a moderator

Re: How to read a file in zip or gz format usin [message #2648 is a reply to message #2507]

Thu, 04 August 1994 21:42

oet
Messages: 28
Registered: February 1993

Junior Member

>
> I am asking for help to read a file in zip or gz format using IDL. At the
> moment, I have a huge data file in ASCII format about 10 Mbytes, it thus
> takes a long time to read in all the data. I wonder it may take less time if
> the data is compressed using either zip or gzip methods. But how can I read
> such compressed file? Any advice over there is highly appreciated.
>
>

No problem, I had a similiar problem some time ago, here is the data read
procedure I used,

Thomas Oettli

FUNCTION GET_RC, f

; Return reference coordinates

coords = { x_ref:fltarr(305,269), y_ref:fltarr(305,269) }
spawn, ['gunzip','-c',f], UNIT=lun, /NOSHELL

readu, lun, coords
free_lun, lun
return, coords

END

Report message to a moderator

Previous Topic:	Diagnostic printout
Next Topic:	Different shifts on rows of matrices

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 22:10:01 PDT 2025

Total time taken to generate the page: 3.27691 seconds