| writing large 3D data file fails [message #68263] |
Wed, 07 October 2009 05:17  |
dorthe
Messages: 12
Registered: April 2007
|
Junior Member |
|
|
One more question:
I have a fltarr of 4008x4008x865 voxels that I'm trying to write to a
file using
GET_LUN, lun
OPENW, lun, '/nfs/blahblah.dat'
WRITEU, lun, volume
CLOSE, lun
FREE_LUN, lun
this normally works like a charm for writing a simple binary data
file, but for this large dataset, I can't get it to work? The file
that get's written is way too small (about 3.5 GB - if I write it as a
netDCF it is ¨21 GB, which is more like the right size)
Any ideas what goes wrong here?
Thanks,
Dorthe
|
|
|
|
| Re: writing large 3D data file fails [message #68291 is a reply to message #68263] |
Wed, 14 October 2009 14:57  |
Mark[1]
Messages: 66
Registered: February 2008
|
Member |
|
|
On Oct 13, 10:58 pm, Dorthe Wildenschild
> Thanks for trying to help, - I really don't know what's wrong, but
> instead of working out the kinks of IDL, I may just write the volume
> as 3 or 4 smaller sections... - a bit sad though.
> Cheers,
> Dorthe
If that's enough to make you sad, you must be new around these parts.
My word, I could tell you stories that could make you weep. I'm green
with envy that you have a system on which you could even consider
writing a 50 GiB file in one go!
WRITEU doesn't add any leading or trailing bytes by default, so you
can write it in as many chunks as you like, as long as you get the
order right. Just loop over the outer (final) dimension.
|
|
|
|
| Re: writing large 3D data file fails [message #68560 is a reply to message #68263] |
Mon, 02 November 2009 17:51 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
A.R. writes:
> So, this will work to write your data to one file, but the next issue
> crops up if you want to re-open it and work with it again. Trying to
> 'readu' a volume with more than 2^31 elements also crashes IDL due to
> a segmentation fault.
>
> If you can figure out how to re-open it, then you'll really make my
> day! Still can't figure that one out . . .
If you can get by reading it a slice at a time
(from anywhere in the stack), then the Associated
Variable method is the way to go. Suppose each
"slice" is a 4000 by 5000 floating point array:
Openr, lun, 'myfile.dat', /Get_Lun
data = Assoc(lun, FltArr(4000,5000))
Now, if you wanted the 35th slice:
slice = data[34]
When you are done with it, just close it.
Free_Lun, lun
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: writing large 3D data file fails [message #68565 is a reply to message #68263] |
Mon, 02 November 2009 14:31 |
penteado
Messages: 866
Registered: February 2018
|
Senior Member
Administrator |
|
|
On Nov 2, 8:17 pm, "A.R." <alrom...@gmail.com> wrote:
> On Oct 14, 1:57 pm, Mark <mark.h...@gmail.com> wrote:
>
>
>
>> On Oct 13, 10:58 pm, Dorthe Wildenschild
>
>>> Thanks for trying to help, - I really don't know what's wrong, but
>>> instead of working out the kinks of IDL, I may just write the volume
>>> as 3 or 4 smaller sections... - a bit sad though.
>>> Cheers,
>>> Dorthe
>
>> If that's enough to make you sad, you must be new around these parts.
>> My word, I could tell you stories that could make you weep. I'm green
>> with envy that you have a system on which you could even consider
>> writing a 50 GiB file in one go!
>
>> WRITEU doesn't add any leading or trailing bytes by default, so you
>> can write it in as many chunks as you like, as long as you get the
>> order right. Just loop over the outer (final) dimension.
>
> Hello,
>
> I had a similar problem that I posted to an earlier discussion:http://groups.google.com/group/comp.lang.idl-pvwa ve/browse_thread/thr...
>
> I was also trying to writeu to a volume with larger than 2^31
> elements, during which IDL quit due to a segmentation fault. The best
> answer I got was that it's just something finnicky with IDL, and that
> I should write a bug report to ITTVIS. As a workaround, I wrote a
> small piece of code which writes my volume in individual slices to one
> large file:
>
> pro write_bindatxl,fname,volume, nz
> close,10
> openw,10,fname
> for i = 0,(nz-1) do writeu,10,volume[*,*,i]
> close,10
> end
>
> So, this will work to write your data to one file, but the next issue
> crops up if you want to re-open it and work with it again. Trying to
> 'readu' a volume with more than 2^31 elements also crashes IDL due to
> a segmentation fault.
>
> If you can figure out how to re-open it, then you'll really make my
> day! Still can't figure that one out . . .
Why can't you read it with readu the same way you used writeu?
|
|
|
|
| Re: writing large 3D data file fails [message #68566 is a reply to message #68291] |
Mon, 02 November 2009 14:17 |
A.R.
Messages: 3
Registered: September 2009
|
Junior Member |
|
|
On Oct 14, 1:57 pm, Mark <mark.h...@gmail.com> wrote:
> On Oct 13, 10:58 pm, Dorthe Wildenschild
>
>> Thanks for trying to help, - I really don't know what's wrong, but
>> instead of working out the kinks of IDL, I may just write the volume
>> as 3 or 4 smaller sections... - a bit sad though.
>> Cheers,
>> Dorthe
>
> If that's enough to make you sad, you must be new around these parts.
> My word, I could tell you stories that could make you weep. I'm green
> with envy that you have a system on which you could even consider
> writing a 50 GiB file in one go!
>
> WRITEU doesn't add any leading or trailing bytes by default, so you
> can write it in as many chunks as you like, as long as you get the
> order right. Just loop over the outer (final) dimension.
Hello,
I had a similar problem that I posted to an earlier discussion:
http://groups.google.com/group/comp.lang.idl-pvwave/browse_t hread/thread/c236bbf7a54085ee/0952d20b84caae00?lnk=gst&q =segmentation+fault#0952d20b84caae00
I was also trying to writeu to a volume with larger than 2^31
elements, during which IDL quit due to a segmentation fault. The best
answer I got was that it's just something finnicky with IDL, and that
I should write a bug report to ITTVIS. As a workaround, I wrote a
small piece of code which writes my volume in individual slices to one
large file:
pro write_bindatxl,fname,volume, nz
close,10
openw,10,fname
for i = 0,(nz-1) do writeu,10,volume[*,*,i]
close,10
end
So, this will work to write your data to one file, but the next issue
crops up if you want to re-open it and work with it again. Trying to
'readu' a volume with more than 2^31 elements also crashes IDL due to
a segmentation fault.
If you can figure out how to re-open it, then you'll really make my
day! Still can't figure that one out . . .
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
