| Re: reading data-files from archive? [message #56213] |
Mon, 15 October 2007 00:38 |
![Maarten[1] is currently offline](theme/default/images/xoffline.png.pagespeed.ic.XRkd1fkXye.png "Maarten[1] is currently offline")
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
On Oct 13, 9:58 am, Andrew_Cool <poo...@gmail.com> wrote:
> On Oct 12, 7:03 pm, Sven Utcke <utcke+n...@informatik.uni-hamburg.de>
> wrote:
>
>> is IDL capable of reading data out of archive-files? What I mean by
>> that, given a somfile.zip or somefile.arc, could I somehow access the
>> /some/path/to/image.tiff stored inside the somfile.zip? If not, would
>> it be easy to add this functionality?
>
> Try the COMPRESS keyword to the Open command??
That will allow transparent reading of gzip files. A gzip file is not
an archive, as the file can only hold a single uncompressed file. zip
files (or tar archive with external compression: .tar.gz, .tar.bz2)
will hold a larger number of files. It is this latter capability the
OP is after. Without shell supprt, I don't think IDL can do this out
of the box. The code would be platform dependent (it would not be too
bad on *nix, I can't comment on running it on Windows).
Maarten
|
|
|
|
| Re: reading data-files from archive? [message #56227 is a reply to message #56213] |
Sat, 13 October 2007 02:58  |
pook41
Messages: 16
Registered: January 2007
|
Junior Member |
|
|
On Oct 12, 7:03 pm, Sven Utcke <utcke+n...@informatik.uni-hamburg.de>
wrote:
> Hi,
>
> I'm a complete and utter newbie to IDL, so please bear with me:
>
> is IDL capable of reading data out of archive-files? What I mean by
> that, given a somfile.zip or somefile.arc, could I somehow access the
> /some/path/to/image.tiff stored inside the somfile.zip? If not, would
> it be easy to add this functionality?
>
> Background: we're doing tomography here, and expect to create some PB
> of data each year, starting late 2008/early 2009. Unfortunately the
> only system able to deal with that much data (dCache) is only well
> suited for files between 100MB and 2GB, i.e. too big for a single
> slice, and too small for a stack of slices. Using archives would be
> one way around this limitation...
>
> Any pointers appreciated
>
> Sven
Try the COMPRESS keyword to the Open command??
Cheers,
Andrew
|
|
|
|
| Re: reading data-files from archive? [message #56289 is a reply to message #56227] |
Fri, 12 October 2007 04:11 |
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
On Oct 12, 10:03 am, Sven Utcke <utcke+n...@informatik.uni-hamburg.de>
wrote:
> is IDL capable of reading data out of archive-files? What I mean by
> that, given a somfile.zip or somefile.arc, could I somehow access the
> /some/path/to/image.tiff stored inside the somfile.zip? If not, would
> it be easy to add this functionality?
While the file routines will deal with individual gzipped files on the
fly, I'm not aware of the ability to use zip, arc or tar archives on
the fly. Of course you can call the system tools (spawn & friends) to
unzip/untar the data in a temporary location, but I can see why you'd
want to avoid that.
> Background: we're doing tomography here, and expect to create some PB
> of data each year, starting late 2008/early 2009. Unfortunately the
> only system able to deal with that much data (dCache) is only well
> suited for files between 100MB and 2GB, i.e. too big for a single
> slice, and too small for a stack of slices. Using archives would be
> one way around this limitation...
I think you'll be better off using scientific data formats, in
particular hdf. The files can be up to 2 GB *hdf4) or way larger
(hdf5). Note that the formats are very different: hdf4 has data types
for image data, etc., while hdf5 basically stores arrays. In my
opinion hdf4 was over-engineered. For storing multiple items under
potentially the same name (but a different location within the file),
you're better off with hdf5.
It is easy to add metadata to the data-sets in hdf, although I would
advise to duplicate at least some of the metadata in a database for
easier retrieval later on.
Maarten
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
