Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
Re: time cost for file reading increasing!!! same file!
On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote: > Hi everyone, > I meet a mysterious problem on IDL file reading. > The problem is as following: > I just read the same hdf5 file seveal times and see the time costing. > I find the first 270 file reading is very quickly but after that, the > reading speed drops down although the file is the same one. > > The simple code is here: > pro try > filename='./particles.h5' > file_id=h5f_open(filename) > for i=1,2 do begin > i=1 ; here just for test, so it will > always read > print,'Systime=',Systime(0) ; here output the systime > dataset_name='Electrons_at_PE_302' > dataset_id=h5d_open(file_id,dataset_name) > ptcl0=h5d_read(dataset_id) > h5d_close,dataset_id > endfor > h5f_close,file_id > end > > I have check the memory, it does not increase. So the memory should no > problem. > Why the reading time increase? I am also sure there is no problem with > the hard disk heating. > So, what's the problem? What does your system memory look like? I don't know anything about h5d_read, but if there is a memory leak, you could be running low on RAM. Try adding in a line like help,/memory to the interior of the loop. None of the numbers should increase over time unless you are leaking memory. chris
I use: mem=memory(/current) print,mem inside the loop, and the 'print,mem' always the same value and I also use the top command in the system to check the occupied memory, it does not change. On Feb 8, 4:58 pm, Chris <beaum...@ifa.hawaii.edu> wrote: > On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote: > > > > > >> Hi everyone, >> I meet a mysterious problem on IDL file reading. >> The problem is as following: >> I just read the same hdf5 file seveal times and see the time costing. >> I find the first 270 file reading is very quickly but after that, the >> reading speed drops down although the file is the same one. > >> The simple code is here: >> pro try >> filename='./particles.h5' >> file_id=h5f_open(filename) >> for i=1,2 do begin >> i=1 ; here just for test, so it will >> always read >> print,'Systime=',Systime(0) ; here output the systime >> dataset_name='Electrons_at_PE_302' >> dataset_id=h5d_open(file_id,dataset_name) >> ptcl0=h5d_read(dataset_id) >> h5d_close,dataset_id >> endfor >> h5f_close,file_id >> end > >> I have check the memory, it does not increase. So the memory should no >> problem. >> Why the reading time increase? I am also sure there is no problem with >> the hard disk heating. >> So, what's the problem? > > What does your system memory look like? I don't know anything about > h5d_read, but if there is a memory leak, you could be running low on > RAM. Try adding in a line like > help,/memory > to the interior of the loop. None of the numbers should increase over > time unless you are leaking memory. > > chris- Hide quoted text - > > - Show quoted text -
The help,/memory output is like this: j= 87Systime=Mon Feb 8 17:11:28 2010Memory= 2762878 heap memory used: 2762878, max: 4706734, gets: 9133, frees: 8314 j= 88Systime=Mon Feb 8 17:11:29 2010Memory= 2762878 heap memory used: 2762878, max: 4706734, gets: 9153, frees: 8334 j= 89Systime=Mon Feb 8 17:11:29 2010Memory= 2762878 So the heap memory used does not change.
On Feb 8, 8:05 pm, minchen <mchen...@gmail.com> wrote: > I use: > mem=memory(/current) > print,mem > > inside the loop, and the 'print,mem' always the same value > and I also use the top command in the system to check the occupied > memory, it does not change. > > On Feb 8, 4:58 pm, Chris <beaum...@ifa.hawaii.edu> wrote: > >> On Feb 8, 1:20 pm, minchen <mchen...@gmail.com> wrote: > >>> Hi everyone, >>> I meet a mysterious problem on IDL file reading. >>> The problem is as following: >>> I just read the same hdf5 file seveal times and see the time costing. >>> I find the first 270 file reading is very quickly but after that, the >>> reading speed drops down although the file is the same one. > >>> The simple code is here: >>> pro try >>> filename='./particles.h5' >>> file_id=h5f_open(filename) >>> for i=1,2 do begin >>> i=1 ; here just for test, so it will >>> always read >>> print,'Systime=',Systime(0) ; here output the systime >>> dataset_name='Electrons_at_PE_302' >>> dataset_id=h5d_open(file_id,dataset_name) >>> ptcl0=h5d_read(dataset_id) >>> h5d_close,dataset_id >>> endfor >>> h5f_close,file_id >>> end > >>> I have check the memory, it does not increase. So the memory should no >>> problem. >>> Why the reading time increase? I am also sure there is no problem with >>> the hard disk heating. >>> So, what's the problem? > >> What does your system memory look like? I don't know anything about >> h5d_read, but if there is a memory leak, you could be running low on >> RAM. Try adding in a line like >> help,/memory >> to the interior of the loop. None of the numbers should increase over >> time unless you are leaking memory. > >> chris- Hide quoted text - > >> - Show quoted text - > > Is this a local disk, or a network drive? Is it local but shared by other users? Your problem could potentially be caused by network traffic or other folks hitting the same drive. r
It is a network drive. But it is unlikely the problem of confliction with others. Because I run it several times,every time the reading speed drops down after reading about 280 times of the file.
On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote: > Hi everyone, > I meet a mysterious problem on IDL file reading. > The problem is as following: > I just read the same hdf5 file seveal times and see the time costing. > I find the first 270 file reading is very quickly but after that, the > reading speed drops down although the file is the same one. I am reaching here, but maybe the problem is similar to one discussed in an earlier thread ( http://tinyurl.com/yzrvd96 ) where the correct question should be "Why is the file reading so fast to start with?" Those fast times are deceptive because the IDL thinks the I/ O operation is complete after it has passed the data to the memory cache, but the OS has not yet written the data to disk. Once the memory cache fills, then IDL has to wait until the data is written to the mechanical disk, but those longer times are a better measure of the time to write the data to disk. --Wayne
On second thought, the OP is only reading, not writing, to disk, so the memory cache may not be relevant. On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote: > On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote: > >> Hi everyone, >> I meet a mysterious problem on IDL file reading. >> The problem is as following: >> I just read the same hdf5 file seveal times and see the time costing. >> I find the first 270 file reading is very quickly but after that, the >> reading speed drops down although the file is the same one. > > I am reaching here, but maybe the problem is similar to one discussed > in an earlier thread (http://tinyurl.com/yzrvd96) where the correct > question should be "Why is the file reading so fast to start > with?" Those fast times are deceptive because the IDL thinks the I/ > O operation is complete after it has passed the data to the memory > cache, but the OS has not yet written the data to disk. Once the > memory cache fills, then IDL has to wait until the data is written to > the mechanical disk, but those longer times are a better measure of > the time to write the data to disk. > > --Wayne
I think this is not the reason. Since the speed is slow down with time. At later time, it may cost 10 minutes to read a file. At fist is less than 0.1s. On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote: > On second thought, the OP is only reading, not writing, to disk, so > the memory cache may not be relevant. > > On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote: > > > >> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote: > >>> Hi everyone, >>> I meet a mysterious problem on IDL file reading. >>> The problem is as following: >>> I just read the same hdf5 file seveal times and see the time costing. >>> I find the first 270 file reading is very quickly but after that, the >>> reading speed drops down although the file is the same one. > >> I am reaching here, but maybe the problem is similar to one discussed >> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct >> question should be "Why is the file reading so fast to start >> with?" Those fast times are deceptive because the IDL thinks the I/ >> O operation is complete after it has passed the data to the memory >> cache, but the OS has not yet written the data to disk. Once the >> memory cache fills, then IDL has to wait until the data is written to >> the mechanical disk, but those longer times are a better measure of >> the time to write the data to disk. > >> --Wayne- Hide quoted text - > > - Show quoted text -
On Feb 9, 1:22 pm, minchen <mchen...@gmail.com> wrote: > I think this is not the reason. Since the speed is slow down with > time. At later time, it may cost 10 minutes to read a file. At fist is > less than 0.1s. > > On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote: > >> On second thought, the OP is only reading, not writing, to disk, so >> the memory cache may not be relevant. > >> On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote: > >>> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote: > >>>> Hi everyone, >>>> I meet a mysterious problem on IDL file reading. >>>> The problem is as following: >>>> I just read the same hdf5 file seveal times and see the time costing. >>>> I find the first 270 file reading is very quickly but after that, the >>>> reading speed drops down although the file is the same one. > >>> I am reaching here, but maybe the problem is similar to one discussed >>> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct >>> question should be "Why is the file reading so fast to start >>> with?" Those fast times are deceptive because the IDL thinks the I/ >>> O operation is complete after it has passed the data to the memory >>> cache, but the OS has not yet written the data to disk. Once the >>> memory cache fills, then IDL has to wait until the data is written to >>> the mechanical disk, but those longer times are a better measure of >>> the time to write the data to disk. > >>> --Wayne- Hide quoted text - > >> - Show quoted text - Which version of IDL are you using?
Both 7.0 and 7.1 are tried. On Feb 10, 6:00 am, Bennett <juggernau...@gmail.com> wrote: > On Feb 9, 1:22 pm, minchen <mchen...@gmail.com> wrote: > > > > > >> I think this is not the reason. Since the speed is slow down with >> time. At later time, it may cost 10 minutes to read a file. At fist is >> less than 0.1s. > >> On Feb 8, 6:04 pm, wlandsman <wlands...@gmail.com> wrote: > >>> On second thought, the OP is only reading, not writing, to disk, so >>> the memory cache may not be relevant. > >>> On Feb 8, 9:00 pm, wlandsman <wlands...@gmail.com> wrote: > >>>> On Feb 8, 6:20 pm, minchen <mchen...@gmail.com> wrote: > >>>> > Hi everyone, >>>> > I meet a mysterious problem on IDL file reading. >>>> > The problem is as following: >>>> > I just read the same hdf5 file seveal times and see the time costing. >>>> > I find the first 270 file reading is very quickly but after that, the >>>> > reading speed drops down although the file is the same one. > >>>> I am reaching here, but maybe the problem is similar to one discussed >>>> in an earlier thread (http://tinyurl.com/yzrvd96) where the correct >>>> question should be "Why is the file reading so fast to start >>>> with?" Those fast times are deceptive because the IDL thinks the I/ >>>> O operation is complete after it has passed the data to the memory >>>> cache, but the OS has not yet written the data to disk. Once the >>>> memory cache fills, then IDL has to wait until the data is written to >>>> the mechanical disk, but those longer times are a better measure of >>>> the time to write the data to disk. > >>>> --Wayne- Hide quoted text - > >>> - Show quoted text - > > Which version of IDL are you using?- Hide quoted text - > > - Show quoted text -