| Re: Trouble writing very large files [message #62622] |
Tue, 30 September 2008 07:41 |
Jeremy Bailin
Messages: 618
Registered: April 2008
|
Senior Member |
|
|
On Sep 29, 12:04 pm, David Fanning <n...@dfanning.com> wrote:
> Vince Hradil writes:
>> FWIW - that worked fine. I was able to write each of the 15 time
>> frames separately:
>
>> openw, lun, oname, /get_lun
>> for f=3D0L, nframes-1 do writeu, lun, array[*,*,*,f]
>> free_lun, lun
>
> Humm. Well, not sure exactly what that tells us,
> but my guess (having tried to do this a time or
> two myself) is that the calculation of some internal
> pointer is off. If my experience is any indication,
> off by 1. :-(
>
> My theory is that by writing the file differently,
> the pointer was calculated differently in the code.
> Good as any, I guess. :-)
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Coyote's Guide to IDL Programming (www.dfanning.com)
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
It might be interesting to know whether an array of the same total
size but with different dimensions works or not.
-Jeremy.
|
|
|
|
| Re: Trouble writing very large files [message #62643 is a reply to message #62622] |
Mon, 29 September 2008 09:04  |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Vince Hradil writes:
> FWIW - that worked fine. I was able to write each of the 15 time
> frames separately:
>
> openw, lun, oname, /get_lun
> for f=3D0L, nframes-1 do writeu, lun, array[*,*,*,f]
> free_lun, lun
Humm. Well, not sure exactly what that tells us,
but my guess (having tried to do this a time or
two myself) is that the calculation of some internal
pointer is off. If my experience is any indication,
off by 1. :-(
My theory is that by writing the file differently,
the pointer was calculated differently in the code.
Good as any, I guess. :-)
Cheers,
David
--
David Fanning, Ph.D.
Coyote's Guide to IDL Programming (www.dfanning.com)
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Trouble writing very large files [message #62645 is a reply to message #62643] |
Mon, 29 September 2008 08:57 |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
On Sep 29, 9:16 am, Vince Hradil <vincehra...@gmail.com> wrote:
> On Sep 29, 8:56 am, David Fanning <n...@dfanning.com> wrote:
>
>
>
>> Vince Hradil writes:
>>> Yes, my solution (for now?) is to write out pieces, then spawn a
>>> "copy /b file1 + file2 ..." I guess I'd like to know how this lun
>>> pointer is working, though...
>
>> I meant, can you write the array, in pieces, into the same
>> file.
>
>> FOR j=0,9 DO BEGIN
>
>> writeU, lun, array[j*start: j*start+chunk]
>
>> ENDFOR
>
>> I just wondered if you could figure out where it started
>> having problems writing.
>
>> Cheers,
>
>> David
>> --
>> David Fanning, Ph.D.
>> Fanning Software Consulting, Inc.
>> Coyote's Guide to IDL Programming:http://www.dfanning.com/
>> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
>
> I'll try it and see what happens.
FWIW - that worked fine. I was able to write each of the 15 time
frames separately:
openw, lun, oname, /get_lun
for f=0L, nframes-1 do writeu, lun, array[*,*,*,f]
free_lun, lun
|
|
|
|
| Re: Trouble writing very large files [message #62646 is a reply to message #62645] |
Mon, 29 September 2008 07:16 |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
On Sep 29, 8:56 am, David Fanning <n...@dfanning.com> wrote:
> Vince Hradil writes:
>> Yes, my solution (for now?) is to write out pieces, then spawn a
>> "copy /b file1 + file2 ..." I guess I'd like to know how this lun
>> pointer is working, though...
>
> I meant, can you write the array, in pieces, into the same
> file.
>
> FOR j=0,9 DO BEGIN
>
> writeU, lun, array[j*start: j*start+chunk]
>
> ENDFOR
>
> I just wondered if you could figure out where it started
> having problems writing.
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
I'll try it and see what happens.
|
|
|
|
| Re: Trouble writing very large files [message #62647 is a reply to message #62646] |
Mon, 29 September 2008 06:56 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Vince Hradil writes:
> Yes, my solution (for now?) is to write out pieces, then spawn a
> "copy /b file1 + file2 ..." I guess I'd like to know how this lun
> pointer is working, though...
I meant, can you write the array, in pieces, into the same
file.
FOR j=0,9 DO BEGIN
writeU, lun, array[j*start: j*start+chunk]
ENDFOR
I just wondered if you could figure out where it started
having problems writing.
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Trouble writing very large files [message #62648 is a reply to message #62647] |
Mon, 29 September 2008 06:51 |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
On Sep 29, 8:42 am, David Fanning <n...@dfanning.com> wrote:
> Vince Hradil writes:
>> IDL crashes when I let it try to write the file. However, I can do
>> the commands that I typed manipulating the pointer. Sorry it wasn't
>> that clear. So, to summarize - The point_lun commands seems to work
>> fine, but the writeu command crashes.
>
> Have you tried writing the array in pieces?
>
> Cheers,
>
> David
> --
> David Fanning, Ph.D.
> Fanning Software Consulting, Inc.
> Coyote's Guide to IDL Programming:http://www.dfanning.com/
> Sepore ma de ni thui. ("Perhaps thou speakest truth.")
Yes, my solution (for now?) is to write out pieces, then spawn a
"copy /b file1 + file2 ..." I guess I'd like to know how this lun
pointer is working, though...
|
|
|
|
| Re: Trouble writing very large files [message #62649 is a reply to message #62648] |
Mon, 29 September 2008 06:42 |
David Fanning
Messages: 11724
Registered: August 2001
|
Senior Member |
|
|
Vince Hradil writes:
> IDL crashes when I let it try to write the file. However, I can do
> the commands that I typed manipulating the pointer. Sorry it wasn't
> that clear. So, to summarize - The point_lun commands seems to work
> fine, but the writeu command crashes.
Have you tried writing the array in pieces?
Cheers,
David
--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.dfanning.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")
|
|
|
|
| Re: Trouble writing very large files [message #62650 is a reply to message #62649] |
Mon, 29 September 2008 06:31 |
Vince Hradil
Messages: 574
Registered: December 1999
|
Senior Member |
|
|
On Sep 27, 5:16 pm, Craig Markwardt
<craigm...@REMOVEcow.physics.wisc.edu> wrote:
> Vince Hradil <vincehra...@gmail.com> writes:
>> I want to write a very large array to a file. I'm trying:
>
>> array = intarr(640,640,1073,15) ;; 13,185,024,000 bytes!
>
>> openw, lun, 'foo.dat', /get_lun
>> writeu, lun, array
>> free_lun, lun
>
>> IDL crashes when the writeu fires up.
>
> Wait, if IDL crashes, how were you able to check the file pointer
> location afterward??? If you mean that IDL reported an error but
> continued to run, what was the error?
>
> Craig
>
>
>
>
>
>> print, !version
>> { x86_64 Win32 Windows Microsoft Windows 7.0 Oct 25 2007 64
>> 64}
>
>> The pointer seems to expand okay:
>
>> IDL> point_lun, -lun, pos
>> IDL> help, pos
>> POS LONG = 0
>> IDL> point_lun, lun, 640ll * 640ll * 1073ll * 15ll * 2ll
>> IDL> point_lun, -lun, pos
>> IDL> help, pos
>> POS LONG64 = 13185024000
>> IDL> point_lun, lun, 0
>> IDL> point_lun, -lun, pos
>> IDL> help, pos
>> POS LONG = 0
>
> --
> ------------------------------------------------------------ --------------
> Craig B. Markwardt, Ph.D. EMAIL: cbmarkwardt+use...@gmail.com
> ------------------------------------------------------------ --------------
IDL crashes when I let it try to write the file. However, I can do
the commands that I typed manipulating the pointer. Sorry it wasn't
that clear. So, to summarize - The point_lun commands seems to work
fine, but the writeu command crashes.
|
|
|
|
| Re: Trouble writing very large files [message #62668 is a reply to message #62650] |
Sat, 27 September 2008 15:16 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
Vince Hradil <vincehradil@gmail.com> writes:
> I want to write a very large array to a file. I'm trying:
>
> array = intarr(640,640,1073,15) ;; 13,185,024,000 bytes!
>
> openw, lun, 'foo.dat', /get_lun
> writeu, lun, array
> free_lun, lun
>
> IDL crashes when the writeu fires up.
Wait, if IDL crashes, how were you able to check the file pointer
location afterward??? If you mean that IDL reported an error but
continued to run, what was the error?
Craig
>
> print, !version
> { x86_64 Win32 Windows Microsoft Windows 7.0 Oct 25 2007 64
> 64}
>
> The pointer seems to expand okay:
>
> IDL> point_lun, -lun, pos
> IDL> help, pos
> POS LONG = 0
> IDL> point_lun, lun, 640ll * 640ll * 1073ll * 15ll * 2ll
> IDL> point_lun, -lun, pos
> IDL> help, pos
> POS LONG64 = 13185024000
> IDL> point_lun, lun, 0
> IDL> point_lun, -lun, pos
> IDL> help, pos
> POS LONG = 0
--
------------------------------------------------------------ --------------
Craig B. Markwardt, Ph.D. EMAIL: cbmarkwardt+usenet@gmail.com
------------------------------------------------------------ --------------
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
