| Re: time cost for file reading increasing!!! same file! [message #69721] |
Wed, 10 February 2010 06:00  |
Juggernaut
Messages: 83
Registered: June 2008
|
Member |
|
|
On Feb 9, 1:22 pm, minchen <mchen...@gmail.com> wrote:
> I think this is not the reason. Since the speed is slow down with
> time. At later time, it may cost 10 minutes to read a file. At fist is
> less than 0.1s.
>
> On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote:
>
>> On second thought, the OP is only reading, not writing, to disk, so
>> the memory cache may not be relevant.
>
>> On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote:
>
>>> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote:
>
>>>> Hi everyone,
>>>> I meet a mysterious problem on IDL file reading.
>>>> The problem is as following:
>>>> I just read the same hdf5 file seveal times and see the time costing.
>>>> I find the first 270 file reading is very quickly but after that, the
>>>> reading speed drops down although the file is the same one.
>
>>> I am reaching here, but maybe the problem is similar to one discussed
>>> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct
>>> question should be "Why is the file reading so fast to start
>>> with?" Those fast times are deceptive because the IDL thinks the I/
>>> O operation is complete after it has passed the data to the memory
>>> cache, but the OS has not yet written the data to disk. Once the
>>> memory cache fills, then IDL has to wait until the data is written to
>>> the mechanical disk, but those longer times are a better measure of
>>> the time to write the data to disk.
>
>>> --Wayne- Hide quoted text -
>
>> - Show quoted text -
Which version of IDL are you using?
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69733 is a reply to message #69721] |
Tue, 09 February 2010 10:22  |
minchen
Messages: 8
Registered: February 2010
|
Junior Member |
|
|
I think this is not the reason. Since the speed is slow down with
time. At later time, it may cost 10 minutes to read a file. At fist is
less than 0.1s.
On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote:
> On second thought, the OP is only reading, not writing, to disk, so
> the memory cache may not be relevant.
>
> On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote:
>
>
>
>> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote:
>
>>> Hi everyone,
>>> I meet a mysterious problem on IDL file reading.
>>> The problem is as following:
>>> I just read the same hdf5 file seveal times and see the time costing.
>>> I find the first 270 file reading is very quickly but after that, the
>>> reading speed drops down although the file is the same one.
>
>> I am reaching here, but maybe the problem is similar to one discussed
>> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct
>> question should be "Why is the file reading so fast to start
>> with?" Those fast times are deceptive because the IDL thinks the I/
>> O operation is complete after it has passed the data to the memory
>> cache, but the OS has not yet written the data to disk. Once the
>> memory cache fills, then IDL has to wait until the data is written to
>> the mechanical disk, but those longer times are a better measure of
>> the time to write the data to disk.
>
>> --Wayne- Hide quoted text -
>
> - Show quoted text -
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69746 is a reply to message #69733] |
Mon, 08 February 2010 18:04 |
wlandsman
Messages: 743
Registered: June 2000
|
Senior Member |
|
|
On second thought, the OP is only reading, not writing, to disk, so
the memory cache may not be relevant.
On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote:
> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote:
>
>> Hi everyone,
>> I meet a mysterious problem on IDL file reading.
>> The problem is as following:
>> I just read the same hdf5 file seveal times and see the time costing.
>> I find the first 270 file reading is very quickly but after that, the
>> reading speed drops down although the file is the same one.
>
> I am reaching here, but maybe the problem is similar to one discussed
> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct
> question should be "Why is the file reading so fast to start
> with?" Those fast times are deceptive because the IDL thinks the I/
> O operation is complete after it has passed the data to the memory
> cache, but the OS has not yet written the data to disk. Once the
> memory cache fills, then IDL has to wait until the data is written to
> the mechanical disk, but those longer times are a better measure of
> the time to write the data to disk.
>
> --Wayne
|
|
|
|
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69749 is a reply to message #69748] |
Mon, 08 February 2010 17:16 |
Robert Moss, PhD
Messages: 29
Registered: November 2006
|
Junior Member |
|
|
On Feb 8, 8:05 pm, minchen <mchen...@gmail.com> wrote:
> I use:
> mem=memory(/current)
> print,mem
>
> inside the loop, and the 'print,mem' always the same value
> and I also use the top command in the system to check the occupied
> memory, it does not change.
>
> On Feb 8, 4:58 pm, Chris <beaum...@ifa.hawaii.edu> wrote:
>
>> On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote:
>
>>> Hi everyone,
>>> I meet a mysterious problem on IDL file reading.
>>> The problem is as following:
>>> I just read the same hdf5 file seveal times and see the time costing.
>>> I find the first 270 file reading is very quickly but after that, the
>>> reading speed drops down although the file is the same one.
>
>>> The simple code is here:
>>> pro try
>>> filename='./particles.h5'
>>> file_id=h5f_open(filename)
>>> for i=1,2 do begin
>>> i=1 ; here just for test, so it will
>>> always read
>>> print,'Systime=',Systime(0) ; here output the systime
>>> dataset_name='Electrons_at_PE_302'
>>> dataset_id=h5d_open(file_id,dataset_name)
>>> ptcl0=h5d_read(dataset_id)
>>> h5d_close,dataset_id
>>> endfor
>>> h5f_close,file_id
>>> end
>
>>> I have check the memory, it does not increase. So the memory should no
>>> problem.
>>> Why the reading time increase? I am also sure there is no problem with
>>> the hard disk heating.
>>> So, what's the problem?
>
>> What does your system memory look like? I don't know anything about
>> h5d_read, but if there is a memory leak, you could be running low on
>> RAM. Try adding in a line like
>> help,/memory
>> to the interior of the loop. None of the numbers should increase over
>> time unless you are leaking memory.
>
>> chris- Hide quoted text -
>
>> - Show quoted text -
>
>
Is this a local disk, or a network drive? Is it local but shared by
other users? Your problem could potentially be caused by network
traffic or other folks hitting the same drive.
r
|
|
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69751 is a reply to message #69750] |
Mon, 08 February 2010 17:05 |
minchen
Messages: 8
Registered: February 2010
|
Junior Member |
|
|
I use:
mem=memory(/current)
print,mem
inside the loop, and the 'print,mem' always the same value
and I also use the top command in the system to check the occupied
memory, it does not change.
On Feb 8, 4:58 pm, Chris <beaum...@ifa.hawaii.edu> wrote:
> On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote:
>
>
>
>
>
>> Hi everyone,
>> I meet a mysterious problem on IDL file reading.
>> The problem is as following:
>> I just read the same hdf5 file seveal times and see the time costing.
>> I find the first 270 file reading is very quickly but after that, the
>> reading speed drops down although the file is the same one.
>
>> The simple code is here:
>> pro try
>> filename='./particles.h5'
>> file_id=h5f_open(filename)
>> for i=1,2 do begin
>> i=1 ; here just for test, so it will
>> always read
>> print,'Systime=',Systime(0) ; here output the systime
>> dataset_name='Electrons_at_PE_302'
>> dataset_id=h5d_open(file_id,dataset_name)
>> ptcl0=h5d_read(dataset_id)
>> h5d_close,dataset_id
>> endfor
>> h5f_close,file_id
>> end
>
>> I have check the memory, it does not increase. So the memory should no
>> problem.
>> Why the reading time increase? I am also sure there is no problem with
>> the hard disk heating.
>> So, what's the problem?
>
> What does your system memory look like? I don't know anything about
> h5d_read, but if there is a memory leak, you could be running low on
> RAM. Try adding in a line like
> help,/memory
> to the interior of the loop. None of the numbers should increase over
> time unless you are leaking memory.
>
> chris- Hide quoted text -
>
> - Show quoted text -
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69752 is a reply to message #69751] |
Mon, 08 February 2010 16:58 |
Chris[6]
Messages: 84
Registered: July 2008
|
Member |
|
|
On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote:
> Hi everyone,
> I meet a mysterious problem on IDL file reading.
> The problem is as following:
> I just read the same hdf5 file seveal times and see the time costing.
> I find the first 270 file reading is very quickly but after that, the
> reading speed drops down although the file is the same one.
>
> The simple code is here:
> pro try
> filename='./particles.h5'
> file_id=h5f_open(filename)
> for i=1,2 do begin
> i=1 ; here just for test, so it will
> always read
> print,'Systime=',Systime(0) ; here output the systime
> dataset_name='Electrons_at_PE_302'
> dataset_id=h5d_open(file_id,dataset_name)
> ptcl0=h5d_read(dataset_id)
> h5d_close,dataset_id
> endfor
> h5f_close,file_id
> end
>
> I have check the memory, it does not increase. So the memory should no
> problem.
> Why the reading time increase? I am also sure there is no problem with
> the hard disk heating.
> So, what's the problem?
What does your system memory look like? I don't know anything about
h5d_read, but if there is a memory leak, you could be running low on
RAM. Try adding in a line like
help,/memory
to the interior of the loop. None of the numbers should increase over
time unless you are leaking memory.
chris
|
|
|
|
| Re: time cost for file reading increasing!!! same file! [message #69808 is a reply to message #69721] |
Thu, 11 February 2010 08:56 |
minchen
Messages: 8
Registered: February 2010
|
Junior Member |
|
|
Both 7.0 and 7.1 are tried.
On Feb 10, 6:00 am, Bennett <juggernau...@gmail.com> wrote:
> On Feb 9, 1:22 pm, minchen <mchen...@gmail.com> wrote:
>
>
>
>
>
>> I think this is not the reason. Since the speed is slow down with
>> time. At later time, it may cost 10 minutes to read a file. At fist is
>> less than 0.1s.
>
>> On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote:
>
>>> On second thought, the OP is only reading, not writing, to disk, so
>>> the memory cache may not be relevant.
>
>>> On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote:
>
>>>> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote:
>
>>>> > Hi everyone,
>>>> > I meet a mysterious problem on IDL file reading.
>>>> > The problem is as following:
>>>> > I just read the same hdf5 file seveal times and see the time costing.
>>>> > I find the first 270 file reading is very quickly but after that, the
>>>> > reading speed drops down although the file is the same one.
>
>>>> I am reaching here, but maybe the problem is similar to one discussed
>>>> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct
>>>> question should be "Why is the file reading so fast to start
>>>> with?" Those fast times are deceptive because the IDL thinks the I/
>>>> O operation is complete after it has passed the data to the memory
>>>> cache, but the OS has not yet written the data to disk. Once the
>>>> memory cache fills, then IDL has to wait until the data is written to
>>>> the mechanical disk, but those longer times are a better measure of
>>>> the time to write the data to disk.
>
>>>> --Wayne- Hide quoted text -
>
>>> - Show quoted text -
>
> Which version of IDL are you using?- Hide quoted text -
>
> - Show quoted text -
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
