| large array [message #41312] |
Tue, 12 October 2004 07:52 |
Wolf Schweitzer
Messages: 21
Registered: October 2001
|
Junior Member |
|
|
I need to read a file that is ca. 7 GB large (the file size is defineda
s 2048 x 2048 x 900 are the dimensions, x 2 (integers) + 512 bytes header).
The format is known (.ISQ) and I have a routine that deals with the
header information efficiently; I have some similarly formatted files of
the same scan process around 800-900 MB that I can read without problems.
Generally, I had no problems reading files up to 1.4 GB in size directly
into one variable under IDL.
The data is read into an intarr (2048,2048,900) like this:
imagearray = assoc (.., intarr(...), headersize)
imagearray = temporary (imagearray [0])
Now, the reading takes a little while but it seems to work alright. The
machine - most of the time - does not crash (12 GB RAM, 64-bit AIX
version of IDL 6.1, IBM-Workstation).
However, the visualisation of slice subscripts to this array later does
not display any interesting information; instead, the images of these
large data files look different each time and they do not reflect the
content of the data.
Question:
What do I need to know in order to set up an array for very large data?
Is there a basic difference between arrays < 2 GB and arrays > GB of
size? Are the subscript variables all multiplied before the array is
really looked at, so do all of the subscript variables need to be
Long64? Such as intarr (long64(x),long64(y),long64(z))?
Thanks,
Wolf.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
