| Re: What's better: 1 big HDF file or several samller ones?? [message #32231 is a reply to message #32226] |
Wed, 25 September 2002 05:23  |
Robert Stockwell
Messages: 74
Registered: October 2001
|
Member |
|
|
Brian Huether wrote:
> I have radar data that is broken down based on target orientation (i.e. the
> data is split into 72 5 degree azimuthal windows). So do I create one HDF
> file with 72 datasets? The other thing to consider is this: I need to run
> computational algorithms that will need to access the data so are there
> speed considerations when saving the data this way? Basically I have to data
> storage goals: 1) make the data readily shareable, 2)make the data storage
> appropriate for quick retrieval.
>
> I suppose when it comes to the computational stuff, I can use the one big
> file and then one time I can just read all the info into an array. So the
> awkwardness of the big file will only be problematic one time.
>
> Any thoughts?
>
> -brian
>
>
Hi Brian,
I tend to vote with "one big file". It is easier to keep track of, to backup
(assuming it fits on your media), and to transfer. With many smaller files,
it is not always easy to tell that one of them "disappear".
In fact I did have that problem with my satellite data. It had 13,000 files
in a directory, and retreiving the directory listing resulted in some
100 random files getting dropped (on a slightly older version of linux, 7.0 maybe).
It is very difficult to find an error like that, which would not happen if the
data was in giant files.
I store global data maps (x,y,t) in large (1 gig) yearly files as contiguous
spatial maps one on top of each other, but I also reproduce the data as a
collection of contiguous time series, so depending on what type of data you
want (a snapshot of the data at a particular time, or all the data from one
site as a time series) the retrieval is blazingly fast.
Cheers,
bob
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
