| Processing about 10,000 tif files [message #90976] |
Tue, 19 May 2015 14:07  |
livingwater000
Messages: 2
Registered: May 2015
|
Junior Member |
|
|
Dear Gurus,
I have RGB images in tif format and want to do simple arithmetic calculations.
A big obstacle to me is number of files to be processed, about 10,000 images.
Name of the image files has sequential id such as img_1.tif, img_2.tif, ....., img_10000.tif.
And I have a text file containing image names, coefficients, and offsets.
I think reading the text file as an array with three columns and lines matching number of files seems to be successful.
I've tried to read image files with dialog_pickfile, but I've learned that the function limits byte length and can't seem to go beyond more than 2,000 files or so.
And I've tried to read image files with file_search, but it didn't do calculation in sequential order.
Would you give some suggestion how to read image files in order, so that following calculation should be in order from 1 to 10,000?
Or a way to do calculation with image name corresponding lines in the text file?
Thanks in advance,
Kim
|
|
|
|
| Re: Processing about 10,000 tif files [message #90977 is a reply to message #90976] |
Tue, 19 May 2015 15:03  |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
On Tuesday, May 19, 2015 at 11:07:39 PM UTC+2, livingw...@gmail.com wrote:
> Dear Gurus,
>
> I have RGB images in tif format and want to do simple arithmetic calculations.
> A big obstacle to me is number of files to be processed, about 10,000 images.
> Name of the image files has sequential id such as img_1.tif, img_2.tif, ....., img_10000.tif.
> And I have a text file containing image names, coefficients, and offsets.
> I think reading the text file as an array with three columns and lines matching number of files seems to be successful.
>
> I've tried to read image files with dialog_pickfile, but I've learned that the function limits byte length and can't seem to go beyond more than 2,000 files or so.
> And I've tried to read image files with file_search, but it didn't do calculation in sequential order.
>
> Would you give some suggestion how to read image files in order, so that following calculation should be in order from 1 to 10,000?
> Or a way to do calculation with image name corresponding lines in the text file?
>
> Thanks in advance,
> Kim
I've been there before and I see two ways out of this:
1) construct the file array with something like 'img_'+strtrim(indgen(10000)+1,2)+'.tif'
2) use file_search and then sort the file array to have the order you wish
3) read the text file sequentially and process sequentially
Option 2) is not easy because the files don't have a format like img_00001.tif, otherwise sorting would be easier.
Hope it helps.
Helder
|
|
|
|
| Re: Processing about 10,000 tif files [message #90978 is a reply to message #90977] |
Tue, 19 May 2015 15:12 |
Helder Marchetto
Messages: 520
Registered: November 2011
|
Senior Member |
|
|
On Wednesday, May 20, 2015 at 12:03:23 AM UTC+2, Helder wrote:
> On Tuesday, May 19, 2015 at 11:07:39 PM UTC+2, livingw...@gmail.com wrote:
>> Dear Gurus,
>>
>> I have RGB images in tif format and want to do simple arithmetic calculations.
>> A big obstacle to me is number of files to be processed, about 10,000 images.
>> Name of the image files has sequential id such as img_1.tif, img_2.tif, ....., img_10000.tif.
>> And I have a text file containing image names, coefficients, and offsets.
>> I think reading the text file as an array with three columns and lines matching number of files seems to be successful.
>>
>> I've tried to read image files with dialog_pickfile, but I've learned that the function limits byte length and can't seem to go beyond more than 2,000 files or so.
>> And I've tried to read image files with file_search, but it didn't do calculation in sequential order.
>>
>> Would you give some suggestion how to read image files in order, so that following calculation should be in order from 1 to 10,000?
>> Or a way to do calculation with image name corresponding lines in the text file?
>>
>> Thanks in advance,
>> Kim
>
> I've been there before and I see two ways out of this:
> 1) construct the file array with something like 'img_'+strtrim(indgen(10000)+1,2)+'.tif'
> 2) use file_search and then sort the file array to have the order you wish
> 3) read the text file sequentially and process sequentially
>
> Option 2) is not easy because the files don't have a format like img_00001.tif, otherwise sorting would be easier.
>
> Hope it helps.
> Helder
Brilliant... two ways out in three points. Don't drink and type.
Helder
|
|
|
|
| Re: Processing about 10,000 tif files [message #90979 is a reply to message #90978] |
Tue, 19 May 2015 18:01 |
livingwater000
Messages: 2
Registered: May 2015
|
Junior Member |
|
|
On Tuesday, May 19, 2015 at 6:12:53 PM UTC-4, Helder wrote:
> On Wednesday, May 20, 2015 at 12:03:23 AM UTC+2, Helder wrote:
>> On Tuesday, May 19, 2015 at 11:07:39 PM UTC+2, livingw...@gmail.com wrote:
>>> Dear Gurus,
>>>
>>> I have RGB images in tif format and want to do simple arithmetic calculations.
>>> A big obstacle to me is number of files to be processed, about 10,000 images.
>>> Name of the image files has sequential id such as img_1.tif, img_2.tif, ....., img_10000.tif.
>>> And I have a text file containing image names, coefficients, and offsets.
>>> I think reading the text file as an array with three columns and lines matching number of files seems to be successful.
>>>
>>> I've tried to read image files with dialog_pickfile, but I've learned that the function limits byte length and can't seem to go beyond more than 2,000 files or so.
>>> And I've tried to read image files with file_search, but it didn't do calculation in sequential order.
>>>
>>> Would you give some suggestion how to read image files in order, so that following calculation should be in order from 1 to 10,000?
>>> Or a way to do calculation with image name corresponding lines in the text file?
>>>
>>> Thanks in advance,
>>> Kim
>>
>> I've been there before and I see two ways out of this:
>> 1) construct the file array with something like 'img_'+strtrim(indgen(10000)+1,2)+'.tif'
>> 2) use file_search and then sort the file array to have the order you wish
>> 3) read the text file sequentially and process sequentially
>>
>> Option 2) is not easy because the files don't have a format like img_00001.tif, otherwise sorting would be easier.
>>
>> Hope it helps.
>> Helder
>
> Brilliant... two ways out in three points. Don't drink and type.
> Helder
Thanks much. will try it shortly and update.
Kim
|
|
|
|
| Re: Processing about 10,000 tif files [message #91010 is a reply to message #90976] |
Sun, 24 May 2015 18:23 |
andrewcool777
Messages: 27
Registered: November 2012
|
Junior Member |
|
|
On Wednesday, 20 May 2015 06:37:39 UTC+9:30, livingw...@gmail.com wrote:
> Dear Gurus,
>
> I have RGB images in tif format and want to do simple arithmetic calculations.
> A big obstacle to me is number of files to be processed, about 10,000 images.
> Name of the image files has sequential id such as img_1.tif, img_2.tif, ....., img_10000.tif.
> And I have a text file containing image names, coefficients, and offsets.
> I think reading the text file as an array with three columns and lines matching number of files seems to be successful.
>
> I've tried to read image files with dialog_pickfile, but I've learned that the function limits byte length and can't seem to go beyond more than 2,000 files or so.
> And I've tried to read image files with file_search, but it didn't do calculation in sequential order.
>
> Would you give some suggestion how to read image files in order, so that following calculation should be in order from 1 to 10,000?
> Or a way to do calculation with image name corresponding lines in the text file?
>
> Thanks in advance,
> Kim
How about just rename the files img_00001.tif to img_09999.tif? Solve many problems in a very few lines of code.
And while you're at it, insert some useful coefficients and offsets ino the filenames too.
Andrew
|
|
|
|
| Re: Processing about 10,000 tif files [message #91024 is a reply to message #91010] |
Tue, 26 May 2015 19:05 |
cgguido
Messages: 195
Registered: August 2005
|
Senior Member |
|
|
On Sunday, May 24, 2015 at 8:23:14 PM UTC-5, andrew...@gmail.com wrote:
>
> How about just rename the files img_00001.tif to img_09999.tif? Solve many problems in a very few lines of code.
>
> And while you're at it, insert some useful coefficients and offsets ino the filenames too.
>
> Andrew
Code below will rename all files so that they can be found in order:
f=file_search('img_*.tif', count=n)
max=20000 ;<--- this should be larger than your max
nzeros = ceil(alog10(max)) > 0
myformat = '(i0'+strcompress(nzeros, /re)+')'
for i=0L,n-1 do print, f[i],' -> ', 'img_'+string((strsplit(file_basename(f[i], '.tif'), '_', /ex))[1], format = myformat)+'.tif'
If you like the above, run it like so:
for i=0L,n-1 do file_move, f[i], 'img_'+string((strsplit(file_basename(f[i], '.tif'), '_', /ex))[1], format = myformat)+'.tif'
G
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
