| Trouble writing very large files [message #62676] |
Fri, 26 September 2008 13:12  |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
I want to write a very large array to a file. I'm trying:
array = intarr(640,640,1073,15) ;; 13,185,024,000 bytes!
openw, lun, 'foo.dat', /get_lun
writeu, lun, array
free_lun, lun
IDL crashes when the writeu fires up.
print, !version
{ x86_64 Win32 Windows Microsoft Windows 7.0 Oct 25 2007 64
64}
The pointer seems to expand okay:
IDL> point_lun, -lun, pos
IDL> help, pos
POS LONG = 0
IDL> point_lun, lun, 640ll * 640ll * 1073ll * 15ll * 2ll
IDL> point_lun, -lun, pos
IDL> help, pos
POS LONG64 = 13185024000
IDL> point_lun, lun, 0
IDL> point_lun, -lun, pos
IDL> help, pos
POS LONG = 0
|
|
|
|
| Re: Trouble writing very large files [message #64793 is a reply to message #62676] |
Fri, 16 January 2009 13:21  |
wlandsman
Messages: 743
Registered: June 2000
|
Senior Member |
|
|
On Jan 16, 9:38 am, Maarten <maarten.sn...@knmi.nl> wrote:
> I think the right answer was given by Wayne is his original post. What
> I don't get is why you'd want a 4G unformatted data file in the first
> place, I'd prefer to dump the data to an HDF-5 file.
Well, I encountered the problem when trying to write data in the FITS
data format used in astronomy, and I isolated the problem to the
writeu call used in the IDL FITS writer. You are right that the
problem probably would not occur when writing to HDF-5, since the IDL
HDF writer uses an external library, and not writeu.
As discussed in the earlier thread on this topic, the solution is to
write a big array in pieces, e.g.
for i=0,255 do writeu,1,im[*,*,i]
For the FITS writer, my kluge will be to write a wrapper procedure
writebigu.pro to replace writeu, which will write an array in pieces
if the number of elements exceeds 2^31. --Wayne
|
|
|
|
| Re: Trouble writing very large files [message #64815 is a reply to message #62676] |
Fri, 16 January 2009 06:38 |
Maarten[1]
Messages: 176
Registered: November 2005
|
Senior Member |
|
|
On Jan 16, 1:53 pm, Bennett <juggernau...@gmail.com> wrote:
> There are limitations for file size on different operating systems.
> FAT32 has a limit of 4GB. If you put into IDL 4096UL*4096UL*255UL
> then get your limit. You may be pushing this limit. Not sure what the
> exact problems but this is at least a push in some direction.
Both Bringfried and Wayne are working under Linux, so the file-size
limitation of FAT32 is not likely to be the issue here, not unless the
system is ancient. Given the 64 bit nature of the beast I doubt that.
The file-size limit should be 2^63 bytes.
I think the right answer was given by Wayne is his original post. What
I don't get is why you'd want a 4G unformatted data file in the first
place, I'd prefer to dump the data to an HDF-5 file.
Maarten
|
|
|
|
| Re: Trouble writing very large files [message #64817 is a reply to message #62676] |
Fri, 16 January 2009 04:53 |
Juggernaut
Messages: 83
Registered: June 2008
|
Member |
|
|
On Jan 16, 3:35 am, Bringfried Stecklum <steck...@tls-tautenburg.de>
wrote:
> wlandsman wrote:
>> Vince Hradil posted a message in September about problems writing
>> large files. I am encountering the same problem, but with a
>> variation. Instead of giving a segmentation fault, IDL returns
>> without writing anything.
>
>> IDL> print,!version
>> { x86_64 linux unix linux 7.0 Oct 25 2007 64 64}
>> IDL> im = intarr(4096,4096,256)
>> IDL> openw,1,'test.dat'
>> IDL> writeu,1,im
>> IDL> close,1
>> IDL> $ls -l test.dat
>> -rw-r--r-- 1 wlandsma shadow 0 2009-01-15 15:22 test.dat
>
>> If I instead do the same thing with a 4096 x 4096 x 255 array, then
>> IDL does crash with a segmentation fault. My guess is that an
>> internal IDL counter is not properly defined as 64 bit integer, and
>> so is being set to zero in my first case (where subscripts are a exact
>> multiple of two). In the second case it is being set to a negative
>> number and causing IDL to crash.
>
>> In any case this seems to be an internal IDL bug, since I should be
>> able to write these arrays on a 64 bit machine. --Wayne
>
> It seems to be a writeu bug since save works fine
>
> IDL Version 6.4 (linux x86_64 m64). (c) 2007, ITT Visual Information Solutions
> Installation number: 12207.
> Licensed for use by: TLS Tautenburg
> IDL> hhh=intarr(4096,4096,128)
> IDL> openw,1,'hhh'
> IDL> writeu,1,hhh
> Segmentation fault
>
> 2nd attempt
> IDL> hhh=intarr(4096,4096,128)
> IDL> save,hhh,file='hhh'
> IDL> exit
>
> Regards,
>
> Bringfried
There are limitations for file size on different operating systems.
FAT32 has a limit of 4GB. If you put into IDL 4096UL*4096UL*255UL
then get your limit. You may be pushing this limit. Not sure what the
exact problems but this is at least a push in some direction.
Best of Luck,
Bennett
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
