comp.lang.idl-pvwave archive: archive » point

Home » Public Forums » archive » point_lun is slow

Show: Today's Messages :: Show Polls :: Message Navigator
E-mail to friend

point_lun is slow [message #17437]

Wed, 27 October 1999 00:00

George McCabe
Messages: 23
Registered: June 1997

Junior Member

hello,

reading from a data file at regularly spaced byte locations, 2 bytes at
a time using point_lun - my program is abnormally slow. i don't have
enough experience to guess whether the poor performance is inherent to
point_lun & readu approach or if there are options which are affecting
execution adversely.

i'ld appreciate any thoughts on the topic which might lead to a
solution.

thank you,

george mccabe, gmccabe@replica.gsfc.nasa.gov

Report message to a moderator

Re: point_lun is slow [message #17559 is a reply to message #17437]

Fri, 29 October 1999 00:00

korpela
Messages: 59
Registered: September 1993

Member

In article <381860B1.7145@gsfc.nasa.gov>,
George McCabe <george.mccabe@gsfc.nasa.gov> wrote:
> Struan Gray wrote:
>
>> simplest way to access this data be to use ASSOC to associate a huge
>> array of 2-byte variables with the file and then subscript it as
>> necessary to read in either frames or groups of individual pixels.
>
>
> Struan,
>
> ASSOC - certainly the simplest and also the fastest in cases where every
> element of the data is needed. but when for example only the
> [1538,395778,790018,... 434454018] location elements are needed from the
> file it is faster to point_lun to each. furthermore, the size of each
> associated read can affect the overall speed of a procedure. it's a
> better understanding of this subtlety that i am hoping for.

In this case memory mapping is definitely the way to go. Only accessed
pages get actually get read.

Eric

--
Eric Korpela | An object at rest can never be
korpela@ssl.berkeley.edu | stopped.
<a href="http://sag-www.ssl.berkeley.edu/~korpela">Click for home page.</a>

Report message to a moderator

Re: point_lun is slow [message #17560 is a reply to message #17437]

Fri, 29 October 1999 00:00

korpela
Messages: 59
Registered: September 1993

Member

In article <38170AD4.2EC9@gsfc.nasa.gov>,
George McCabe <george.mccabe@gsfc.nasa.gov> wrote:
> hello,
>
> reading from a data file at regularly spaced byte locations, 2 bytes at
> a time using point_lun - my program is abnormally slow. i don't have
> enough experience to guess whether the poor performance is inherent to
> point_lun & readu approach or if there are options which are affecting
> execution adversely.

I generally map the entire file into memory (see my web site). As long
as it's on local storage its usually faster than reading it into memory,
especially if the file is larger than physical memory.

Currently only works on Unix systems, though. Haven't done a windows
version.

Eric

--
Eric Korpela | An object at rest can never be
korpela@ssl.berkeley.edu | stopped.
<a href="http://sag-www.ssl.berkeley.edu/~korpela">Click for home page.</a>

Report message to a moderator

Re: point_lun is slow [message #17568 is a reply to message #17437]

Thu, 28 October 1999 00:00

Liam Gumley
Messages: 473
Registered: November 1994

Senior Member

George McCabe wrote:
> chunking is certainly much faster, and my algorythm is 'chunking' away
> nicely.
>
> in instances where a small number of the total data elements from the
> file are required, the 'chicken pecking' approach is much faster. but
> when in doubt, chunk.

I believe you can use the 'chunking' method in both cases with high
efficiency. The key here is to access the disk in sequential order, with
as few disk accesses as possible. I'm assuming that the goal is to read
small (say 2 byte) sections of data from the disk at random locations.

The following pseudo-algorithm reads records (chunks) of data from the
disk in sequential order. Only records that cover the specified read
locations are actually read from disk. Each record is only read once.

Sort the array of read locations from lowest to highest
Set the record size to 512 bytes (you can experiment with record sizes)
Set the old record number to -1
Start a loop over the read locations
For this read location, compute the record number in the file
If the record number is different than the old record number
Read the current record
Set the old record number to the current record number
End If
For this read location, compute the byte offset within the record
Extract data from the record at the byte offset
End Loop

This method should be just as efficient for small or large numbers of
read locations.

Cheers,
Liam.

--
Liam E. Gumley
Space Science and Engineering Center, UW-Madison
http://cimss.ssec.wisc.edu/~gumley

Report message to a moderator

Re: point_lun is slow [message #17569 is a reply to message #17437]

Thu, 28 October 1999 00:00

George McCabe
Messages: 23
Registered: June 1997

Junior Member

Struan Gray wrote:

> simplest way to access this data be to use ASSOC to associate a huge
> array of 2-byte variables with the file and then subscript it as
> necessary to read in either frames or groups of individual pixels.

Struan,

ASSOC - certainly the simplest and also the fastest in cases where every
element of the data is needed. but when for example only the
[1538,395778,790018,... 434454018] location elements are needed from the
file it is faster to point_lun to each. furthermore, the size of each
associated read can affect the overall speed of a procedure. it's a
better understanding of this subtlety that i am hoping for.

george

Report message to a moderator

Re: point_lun is slow [message #17571 is a reply to message #17437]

Thu, 28 October 1999 00:00

Struan Gray
Messages: 178
Registered: December 1995

Senior Member

George McCabe, george.mccabe@gsfc.nasa.gov writes:

> reading from a data file at regularly spaced byte
> locations, 2 bytes at a time using point_lun
> - my program is abnormally slow.

Perhaps I'm just being dim (it has been known :-), but wouldn't the
simplest way to access this data be to use ASSOC to associate a huge
array of 2-byte variables with the file and then subscript it as
necessary to read in either frames or groups of individual pixels.

Struan

Report message to a moderator

Re: point_lun is slow [message #17576 is a reply to message #17437]

Thu, 28 October 1999 00:00

<karri
Messages: 4
Registered: October 1999

Junior Member

George,

I am using FibreChannel disks and found out that reading a sector
at a time feels like reading a floppy. With chunk sizes of 256K
I get about 7MB/sec sustained speed. So at least on the Fibre you
have to use large chunks to get any kind of performance.

I also tried 64K chunks but the performance was much worse.

On Wed, 27 Oct 1999, George McCabe wrote:
> do you have a rule of thumb for optimizing chunk size?

--
Regards,

Karri Kaksonen
Picker Nordstar

Report message to a moderator

Re: point_lun is slow [message #17618 is a reply to message #17437]

Mon, 01 November 1999 00:00

George McCabe
Messages: 23
Registered: June 1997

Junior Member

hello and thanks for the reply Liam,

i reason that the record size has to be larger than the seperation
between consecutive extracted elements in order for there to be a gain
in performance. the number of reads is reduced. also reading
sequentially in a chunk is faster than seeking to each datum, an
additional gain.

let's say the ratio of data read to data extracted is 'R'.

(the time it takes to seek and read the extracted data only) divided by
(the time it takes to read sequentially and extract data) is 1 when R
is what value? my guess is 100. what are your thoughts?

george

In article <38187A7E.8E60823A@ssec.wisc.edu>,
Liam Gumley <Liam.Gumley@ssec.wisc.edu> wrote:
> The following pseudo-algorithm reads records (chunks) of data from the
> disk in sequential order. Only records that cover the specified read
> locations are actually read from disk. Each record is only read once.
>
> Sort the array of read locations from lowest to highest
> Set the record size to 512 bytes (you can experiment with record sizes)
> Set the old record number to -1
> Start a loop over the read locations
> For this read location, compute the record number in the file
> If the record number is different than the old record number
> Read the current record
> Set the old record number to the current record number
> End If
> For this read location, compute the byte offset within the record
> Extract data from the record at the byte offset
> End Loop
>
> This method should be just as efficient for small or large numbers of
> read locations.
>
> Cheers,
> Liam.
>
> --
> Liam E. Gumley
> Space Science and Engineering Center, UW-Madison
> http://cimss.ssec.wisc.edu/~gumley

Sent via Deja.com http://www.deja.com/
Before you buy.

Report message to a moderator

Previous Topic:	Is there a way to keep axis text from scaling?
Next Topic:	IDL 5.2.1 -- bug in filled contours?

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

] [

]

Current Time: Wed Oct 08 11:42:23 PDT 2025

Total time taken to generate the page: 0.00496 seconds