comp.lang.idl-pvwave archive: archive

Home » Public Forums » archive » Re: Read Large files

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

Re: Read Large files [message #81229]

Tue, 28 August 2012 11:23

wlandsman
Messages: 743
Registered: June 2000

Senior Member

It should be faster to read as much as your memory will allow in each READ statement. You say that you read "each data record". If each data record is a small fraction of your file size then it will take a long time to read the file. Preferably you can read your entire file in a single READU call.

--Wayne

On Tuesday, August 28, 2012 1:19:03 PM UTC-4, Gompie wrote:
> Hi,
>
> I want to read in 58 gb of data that is spread over several files ( having records)of size ranging from 7.6gb to 1gb.
>
> What is the fastest way to read these data files.
>
> Right now I read in each data record and it takes 6 hrs to do the reading.
>
> Thanks in advance
>
> Gompie

Report message to a moderator

Re: Read Large files [message #81230 is a reply to message #81229]

Tue, 28 August 2012 10:28

David Fanning
Messages: 11724
Registered: August 2001

Senior Member

Gompie writes:

> I want to read in 58 gb of data that is spread over several files ( having records)of size ranging from 7.6gb to 1gb.
> What is the fastest way to read these data files.
> Right now I read in each data record and it takes 6 hrs to do the reading.

I would think the fastest way would be to write a letter
to the Minister of Technology outlining why you need
a new, more powerful machine ASAP.

Cheers,

David

--
David Fanning, Ph.D.
Fanning Software Consulting, Inc.
Coyote's Guide to IDL Programming: http://www.idlcoyote.com/
Sepore ma de ni thui. ("Perhaps thou speakest truth.")

Report message to a moderator

Re: Read Large files [message #81287 is a reply to message #81229]

Fri, 31 August 2012 18:29

Gompie
Messages: 76
Registered: August 2012

Member

Thanks WLandsman,
It works now. I segmented the data into smaller files and removed the possibility of using loops

Report message to a moderator

Previous Topic:	Geodetic functions library GeoDLL 32 Bit and 64 Bit
Next Topic:	very simple but weird problem with CONTOUR

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 15:36:20 PDT 2025

Total time taken to generate the page: 0.00524 seconds