comp.lang.idl-pvwave archive: archive » Re: Writing a very large file

Home » Public Forums » archive » Re: Writing a very large file

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Writing a very large file [message #57411 is a reply to message #57407]

Fri, 07 December 2007 20:50

Kenneth Bowman
Messages: 86
Registered: November 2006

Member

On Dec 7, 2:17 pm, wlandsman <wlands...@gmail.com> wrote:
> I am writing a sequence of images to a single very large file on my
> Linux system. I find that the processing dramatically slows down
> after the first few images. The simplified code looks like the
> following:
>
> So the first four images take ~0.3s each to write, while subsequent
> images require more than 6 seconds each. I suspect that the slowing
> down is due to IDL (or the OS) needing to extend the file size. (I
> checked that it is not a memory usage problem.) So I think
> things would speed up if I could specify the final file size at the
> beginning -- perhaps there is a way to do this in Unix? I have
> experimented with the BUFSIZE and RAWIO keywords to OPENW but so far
> without any improvement.
>
> Thanks for any suggestions, --Wayne

When writing a sequential file like this there should be no need to
pre-allocate the file space.
What I suspect is happening is the following. The first few images go
into cache (memory).
Once the cache space is filled, it starts writing the files to disk,
which is a much slower process.
6 seconds for a 32 MB file isn't great (~5 MB/s), but maybe you have
older hardware or you are writing to
a network volume?

Ken Bowman

Report message to a moderator

[Message index]

		Re: Writing a very large file By: Bringfried Stecklum on Sat, 08 December 2007 08:19
		Re: Writing a very large file By: Kenneth Bowman on Fri, 07 December 2007 20:50
		Re: Writing a very large file By: Jim Pendleton, ITT Vi on Fri, 07 December 2007 13:47
		Re: Writing a very large file By: Jean H. on Fri, 07 December 2007 13:23
		Re: Writing a very large file By: pgrigis on Fri, 07 December 2007 12:39
		Re: Writing a very large file By: David Fanning on Fri, 07 December 2007 12:23
		Re: Writing a very large file By: wlandsman on Mon, 10 December 2007 07:24
		Re: Writing a very large file By: Kenneth Bowman on Mon, 10 December 2007 07:43

Previous Topic:	Re: Version Control Conundrum
Next Topic:	SVN Question

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 20:10:42 PDT 2025

Total time taken to generate the page: 0.00674 seconds