| How to fasten formatted printf? [message #92524] |
Mon, 04 January 2016 18:01  |
enod
Messages: 41
Registered: November 2004
|
Member |
|
|
Hi all,
I usually have over a thousand text files to write. Each file contains 9 columns and several thousand lines. The below is a test.
PRO TEST_PRINTF_FORMATTED_LARGE
ofile='c:\test.txt'
data=DBLARR(9,7000)
t0=SYSTIME(/seconds)
OPENW,fid,ofile,/get_lun
PRINTF,fid,data,format='((3(1x,f10.5),1x,3(1x,f8.2),1x,3(1X, f4.1)))'
FREE_LUN,fid
DELTA_T=SYSTIME(1)-T0*1d0
PRINT,'Total time: '+STRING(DELTA_T,FORMAT='(f)')+' seconds'
END
On my notebook, it takes about 0.14 seconds to finish the above task, e.g.
% Compiled module: TEST_PRINTF_FORMATTED_LARGE.
Total time: 0.1449999809265137 seconds
% Compiled module: TEST_PRINTF_FORMATTED_LARGE.
Total time: 0.1349999904632568 seconds
% Compiled module: TEST_PRINTF_FORMATTED_LARGE.
Total time: 0.1500000953674316 seconds
For 1000 files, it will take over 2 minutes. I wonder whether there is a way to do it very quickly.
Thank you very much!
Yunfeng Tian
|
|
|
|
|
|
| Re: How to fasten formatted printf? [message #92527 is a reply to message #92525] |
Tue, 05 January 2016 08:34  |
Jeremy Bailin
Messages: 618
Registered: April 2008
|
Senior Member |
|
|
On Monday, January 4, 2016 at 9:53:14 PM UTC-5, Craig Markwardt wrote:
> On Monday, January 4, 2016 at 9:01:10 PM UTC-5, Yunfeng Tian wrote:
>> Hi all,
>>
>> I usually have over a thousand text files to write. Each file contains 9 columns and several thousand lines. The below is a test.
> ...
>> For 1000 files, it will take over 2 minutes. I wonder whether there is a way to do it very quickly.
>
> You are probably limited by I/O. You can investigate:
> 1. SSD or hybrid SSD drive
> 2. Some kind of "RAM disk"
> 3. Whether you can satisfy your requirements by writing unformatted binary files with WRITEU instead of formatted binary files with PRINTF.
> 3. Computer with better CPU and I/O throughput performance.
> Other than using binary files, there's really nothing you can do to your IDL code to make it faster.
>
> Craig
If you need it ultimately in ascii, one option is to have IDL write it in binary and then post-process it with a simple C program, which should be able to do the I/O much faster.
-Jeremy.
|
|
|
|
| Re: How to fasten formatted printf? [message #92528 is a reply to message #92524] |
Tue, 05 January 2016 09:57 |
markb77
Messages: 217
Registered: July 2006
|
Senior Member |
|
|
|
Would it improve the speed if you use the STRING procedure (with the /PRINT keyword?) to write your data to a single string variable, and then use printf to simply write that variable to disk? Or, before saving, convert it to byte type and then use WRITEU to save to disk as previously suggested?
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
