| Re: call_external for file I/O [message #13186] |
Mon, 26 October 1998 00:00 |
Charlie Solomon
Messages: 10
Registered: September 1998
|
Junior Member |
|
|
David Foster wrote ...
> I'm wondering if what you are coming up against is a storage order
> problem, in which the elements of the array are being accessed in
> an order that is not optimal given the order of storage of the
> elements in memory. Ideally the dimension that varies fastest in
> memory should be incremented first.
>
> We've run up against this problem before, and have written a C
> routine to optimize access to a 3D array in memory. If you're
> interested in checking it out let me know.
Thank you, yes, I would like to take a look at that.
> You also might want to look into Eric Korpela's VARRAY utility
> that allows you to have memory mapped files:
>
> http://sag-www.ssl.berkeley.edu/~korpela/mmap/
> korpela@islay.ssl.berkeley.edu
VARRAY looks like a good implementation of what our map_file does,
and this site is a very good reference, thank you. I want to work on
doing something like this with WindowsNT. Eric's page says that it
should be straightforward to convert VARRAY to use Win32's...we'll
see!
Thanks, Charlie Solomon
Raytheon Systems Co., El Segundo, CA
|
|
|
|
| Re: call_external for file I/O [message #13190 is a reply to message #13186] |
Mon, 26 October 1998 00:00  |
David Foster
Messages: 341
Registered: January 1996
|
Senior Member |
|
|
Charlie Solomon wrote:
> The data is in a disk file initially, the map_file routine puts the relevant
> part of the file into memory and sets those pages for sequential access, so
> that when IDL accesses the data in memory it is not based on an algorithm
> like most recently used. This is optimal for image data in which you may
> need to access different subsets of the array at different times.
>
> I agree that it is fast, but the method we use on our Solaris machine seems
> much faster for the same files. I don't have time data available right now,
> but I could get it if needed.
Charlie -
I'm wondering if what you are coming up against is a storage order
problem, in which the elements of the array are being accessed in
an order that is not optimal given the order of storage of the
elements in memory. Ideally the dimension that varies fastest in
memory should be incremented first.
We've run up against this problem before, and have written a C
routine to optimize access to a 3D array in memory. If you're
interested in checking it out let me know.
You also might want to look into Eric Korpela's VARRAY utility
that allows you to have memory mapped files:
http://sag-www.ssl.berkeley.edu/~korpela/mmap/
korpela@islay.ssl.berkeley.edu
Dave
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~
David S. Foster Univ. of California, San Diego
Programmer/Analyst Brain Image Analysis Laboratory
foster@bial1.ucsd.edu Department of Psychiatry
(619) 622-5892 8950 Via La Jolla Drive, Suite 2240
La Jolla, CA 92037
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~
|
|
|
|
| Re: call_external for file I/O [message #13213 is a reply to message #13186] |
Thu, 22 October 1998 00:00 |
Charlie Solomon
Messages: 10
Registered: September 1998
|
Junior Member |
|
|
Mark Rivers wrote in message ...
> I am not sure what you mean. Is the data actually in a disk file or is it
in
> memory? If it is in a disk file then some process needs to do the disk
I/O.
> If it is in memory then call_external can be used to get it into IDL.
The data is in a disk file initially, the map_file routine puts the relevant
part of the file into memory and sets those pages for sequential access, so
that when IDL accesses the data in memory it is not based on an algorithm
like most recently used. This is optimal for image data in which you may
need to access different subsets of the array at different times.
> Note that IDL's disk I/O on NT is quite fast. >5MB/sec is not hard to
achieve
> using writeu and readu.
I agree that it is fast, but the method we use on our Solaris machine seems
much faster for the same files. I don't have time data available right now,
but I could get it if needed.
Thanks for your reply,
Charlie Solomon
Raytheon Systems Co.
|
|
|
|
| Re: call_external for file I/O [message #13219 is a reply to message #13213] |
Thu, 22 October 1998 00:00 |
rivers
Messages: 228
Registered: March 1991
|
Senior Member |
|
|
In article <70ldu6$i60@hacgate2.hac.com>, "Charlie Solomon" <crsolomon@west.raytheon.com> writes:
> Hello experts,
>
> Is anyone out there using call_external to read image files into a variable
> on the Windows NT platform? I have used a c routine on the Solaris platform
> called map_file, which bypasses the standard I/O layer and maps data
> directly into/from memory, making data access very fast compared to using
> IDL's built in file I/O utils. I did not write map_file, but my
> understanding is that it utilizes features in unix which allow this sort of
> memory mapping without going through the stdio layer. I would love to get
> the same speed on my NT machine. Any ideas?
I am not sure what you mean. Is the data actually in a disk file or is it in
memory? If it is in a disk file then some process needs to do the disk I/O.
If it is in memory then call_external can be used to get it into IDL.
Note that IDL's disk I/O on NT is quite fast. >5MB/sec is not hard to achieve
using writeu and readu.
____________________________________________________________
Mark Rivers (773) 702-2279 (office)
CARS (773) 702-9951 (secretary)
Univ. of Chicago (773) 702-5454 (FAX)
5640 S. Ellis Ave. (708) 922-0499 (home)
Chicago, IL 60637 rivers@cars.uchicago.edu (e-mail)
or:
Argonne National Laboratory (630) 252-0422 (office)
Building 434A (630) 252-0405 (lab)
9700 South Cass Avenue (630) 252-1713 (beamline)
Argonne, IL 60439 (630) 252-0443 (FAX)
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
