| Rename files using IDL - is it possible? [message #72010] |
Thu, 05 August 2010 00:18  |
LNpellen
Messages: 37
Registered: November 2009
|
Member |
|
|
I have a problem that FILE_DELETE do not work properly on files having
special characters in the filename. The files opens perfectly, but
when I'm finished analysing I want to delete the files automatically
with FILE_DELETE. The files that I'm working with are autogenerated
with (Norwegian) characters in the filenames and I'm pretty sure that
is the reason why FILE_DELETE generates an error.
Any ideas on how I can force IDL to delete these files?
My idea was to rename the file on opening, but I find no FILE_RENAME
function in IDL.
Best regards,
LN
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72072 is a reply to message #72010] |
Thu, 12 August 2010 14:12  |
chris_torrence@NOSPAM
Messages: 528
Registered: March 2007
|
Senior Member |
|
|
Hi all,
Okay, IDL 7.1.2 and IDL 8.0 fix the original poster's problem, with
testAAA.txt (where A is the A with the ring). Using either
DIALOG_PICKFILE or hard-coded, IDL 8.0 will successfully delete the
file.
Now, regarding Heinz's problem - this is unfortunately a pathological
case, where the 195+133 happens to be the UTF-8 encoding for the
extended ASCII character for 197 (the A with the ring).
Internally, IDL tries to recognize UTF-8 strings by the rules for
UTF-8 encoding. If the first byte is > 193, and the second byte is
between 128-191, then IDL assumes that this is a UTF-8 string. The
file I/O routine then converts the filename to UTF-8 encoding, which
converts the 195+133 to 197, and therefore ends up with the same
filename.
Our UTF-8 conversion routines are based upon the UTF-8 standard, and
are designed to the best of our ability to work with multiple
languages and encodings, including European and Asian languages.
Unfortunately, given a random string of bytes, there is no way to
definitively determine that this is a UTF-8 versus native string.
Hopefully, the case given by Heinz will occur rarely if ever.
Cheers,
Chris
ITTVIS
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72096 is a reply to message #72010] |
Tue, 10 August 2010 14:46 |
Heinz Stege
Messages: 189
Registered: January 2003
|
Senior Member |
|
|
On Tue, 10 Aug 2010 04:58:21 -0700 (PDT), LNpellen wrote:
> So the message is:
> Don't use IDL for deleting files that have special characters in the
> name.
> Wait for the next release after 8.0 and hope for a better solution?
I think, this question has to be answered by the ITTVIS folks.
In the case, you are sure that a file with the name filename is
existing, you could do the following check:
if n_elements(file_search(filename)) ge 2 then $
message,'Non-unique file specification...'
So you can stop program execution by issuing an error message, instead
of deleting or renaming wrong files.
Another way could be to check a whole directory for critical file
parallelisms:
a=file_search('*')
if n_elements(uniq(a,sort(a))) ne n_elements(a) then $
message,'Ambiguous file access...'
HTH, Heinz
|
|
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72118 is a reply to message #72010] |
Fri, 06 August 2010 15:48 |
Heinz Stege
Messages: 189
Registered: January 2003
|
Senior Member |
|
|
On Fri, 6 Aug 2010 09:20:20 -0700 (PDT), Chris Torrence wrote:
> Hi LN,
>
> In IDL 7.1 the file I/O routines had some problems with special
> characters. These have all been fixed in IDL 8.0, and if it is
> possible, I would recommend downloading IDL 8.0 and giving that a try.
>
> Hope this helps.
>
> Chris Torrence
> IDL Software Development Manager
> ITTVIS
Hi Chris,
I'm sorry, but it is IDL 8.0 which has a problem. Version { x86 Win32
Windows Microsoft Windows 8.0 Jun 17 2010 32 64}. Here is a
demonstration. It is a little bit complicated, because one can't
create the necessary filenames with IDL 8.0.
1. Create a file with the name test?.txt by use of the Windows
Explorer. The 5th character has to be a "Latin Capital Letter A With
Ring Above". It can be created with keystroke ALT+0197. It is replaced
by a ? in this posting, because it probably would not be transmitted
correctly.
2. Create a 2nd file with the name test??.txt. The 5th character is a
"Latin Capital Letter A With Tilde", keystroke ALT+0195. The 6th
character is a "Horizontal Ellipsis", keystroke ALT+0133.
3. Go to the IDL Workbench and "open" the 2nd file with
file=dialog_pickfile()
4. Delete the 2nd file with: file_delete,file
The first file is deleted instead of the second one. Without any error
message.
Heinz
|
|
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72124 is a reply to message #72010] |
Fri, 06 August 2010 13:16 |
Heinz Stege
Messages: 189
Registered: January 2003
|
Senior Member |
|
|
On Fri, 6 Aug 2010 12:48:58 -0700 (PDT), Paulo Penteado wrote:
> On Aug 6, 4:06�pm, Heinz Stege <public.215....@arcor.de> wrote:
>> did the people in Norway get their new license files already? �In
>> Germany we are still waiting �;-)
>
> That is serious. I got mine, in Brazil, 2.5 hours after I asked for
> it, the day IDL 8 was released.
Okay, to be honest, I must say that I received an evaluation license.
In Germany the support comes from CREASO. After waiting for more than
one week I contacted them, and they gave me a temporary license
immediately. They say that it always takes a long time to get the
license files from ITTVIS.
Heinz
|
|
|
|
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72127 is a reply to message #72010] |
Fri, 06 August 2010 12:02 |
Heinz Stege
Messages: 189
Registered: January 2003
|
Senior Member |
|
|
On Thu, 5 Aug 2010 23:40:15 -0700 (PDT), LNpellen wrote:
> I'm using IDL 7.1 and Windows XP
>
> Now I've tried spawn and FILE_DELETE and OPENR with /DELETE and
> discovered a bit strange behaviour (I think):
>
> The spawn command is either incorrect or not working
> IDL> spawn, 'DEL C:\testaaa.txt' (where the aaa is three norwegian A
> with a ring)
>
> FILE_DELETE and OPENR with /DELETE is not working if the filename is
> defined by dialog_pickfile, but is woring if the filename is defined
> with a hard coded string
>
> Not working if a=dialog_pickfile(), working if a='C:
> \testaaa.txt' (again a is a ring)
> IDL> FILE_DELETE, a, (/NOEXPAND_PATH)
> IDL> OPENR, UNIT, a, /DELETE, /GET_LUN & FREE_LUN, UNIT
>
> I.e. if the string is hardcoded (which I certainly cannot do in my
> program) the commands except spawn works. I did also try a(0) with
> dialog_pickfile to assure it's a string, not an array of strings.
>
> The hard coded A is a copy of the print of A using dialog_pickfile so
> I find no difference in A hard coded or A defined by dialog_pickfile.
>
> Now any suggestions?
Hi LN,
I don't know if you can (or like to) update to Version 8.0 at this
time. So here is what I figured out. In IDL 7.x the strings
a1=dialog_pickfile()
and
a2='C:\testaaa.txt' ; Workbench command line,
; replace a by A_circle!
are different:
print,byte(a1),format='(20z3.2)'
; 43 3a 5c 74 65 73 74 c5 c5 c5 2e 74 78 74
print,byte(a2),format='(20z3.2)'
; 43 3a 5c 74 65 73 74 c3 85 c3 85 c3 85 2e 74 78 74
a1 is the Windows ANSI (or whatever you call it) representation of the
file name, and a2 is UTF-8.
It may be a suitable workaround for you, to change each c5 to c385 in
the byte representation.
By the way, I am running Windows and IDL 7.1.2. In my installation all
three methods (spawn, file_delete, and openr,/delete) are working
without errors for both, a1 and a2. But to be honest, I really don't
know why! How does IDL or Windows know, that "a1 ist ANSI" and "a2 is
UTF-8"?
HTH, Heinz
P.S.: Could it be, that IDL/Windows first tries to treat the file_name
as an UTF-8 string. If it finds a matching file, it deletes it. And if
IDL/Windows does not find a matching file, that the string is
converted from ANSI to UTF-8 for trying to delete that file? Seems to
be a very dangerous story.
|
|
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72138 is a reply to message #72010] |
Thu, 05 August 2010 23:40 |
LNpellen
Messages: 37
Registered: November 2009
|
Member |
|
|
I'm using IDL 7.1 and Windows XP
Now I've tried spawn and FILE_DELETE and OPENR with /DELETE and
discovered a bit strange behaviour (I think):
The spawn command is either incorrect or not working
IDL> spawn, 'DEL C:\testaaa.txt' (where the aaa is three norwegian A
with a ring)
FILE_DELETE and OPENR with /DELETE is not working if the filename is
defined by dialog_pickfile, but is woring if the filename is defined
with a hard coded string
Not working if a=dialog_pickfile(), working if a='C:
\testaaa.txt' (again a is a ring)
IDL> FILE_DELETE, a, (/NOEXPAND_PATH)
IDL> OPENR, UNIT, a, /DELETE, /GET_LUN & FREE_LUN, UNIT
I.e. if the string is hardcoded (which I certainly cannot do in my
program) the commands except spawn works. I did also try a(0) with
dialog_pickfile to assure it's a string, not an array of strings.
The hard coded A is a copy of the print of A using dialog_pickfile so
I find no difference in A hard coded or A defined by dialog_pickfile.
Now any suggestions?
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72149 is a reply to message #72010] |
Thu, 05 August 2010 10:46 |
Craig Markwardt
Messages: 1869
Registered: November 1996
|
Senior Member |
|
|
On Aug 5, 3:18 am, LNpellen <lnpel...@gmail.com> wrote:
> I have a problem that FILE_DELETE do not work properly on files having
> special characters in the filename. The files opens perfectly, but
> when I'm finished analysing I want to delete the files automatically
> with FILE_DELETE. The files that I'm working with are autogenerated
> with (Norwegian) characters in the filenames and I'm pretty sure that
> is the reason why FILE_DELETE generates an error.
>
> Any ideas on how I can force IDL to delete these files?
A trick you can use is to use the /DELETE option for the OPEN
procedure. Then the file is automatically deleted when you close it.
Of course, don't close it until you are done!
This pattern:
OPENR, UNIT, filename, /DELETE, /GET_LUN
FREE_LUN, UNIT
was the cross-platform way people deleted files before FILE_DELETE was
implemented.
Craig
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: Rename files using IDL - is it possible? [message #72202 is a reply to message #72072] |
Fri, 13 August 2010 15:15 |
Heinz Stege
Messages: 189
Registered: January 2003
|
Senior Member |
|
|
Hi Chris.
On Thu, 12 Aug 2010 14:12:05 -0700 (PDT), Chris Torrence wrote:
> Hi all,
>
...
>
> Now, regarding Heinz's problem - this is unfortunately a pathological
> case, where the 195+133 happens to be the UTF-8 encoding for the
> extended ASCII character for 197 (the A with the ring).
>
> Internally, IDL tries to recognize UTF-8 strings by the rules for
> UTF-8 encoding. If the first byte is > 193, and the second byte is
> between 128-191, then IDL assumes that this is a UTF-8 string. The
> file I/O routine then converts the filename to UTF-8 encoding, which
> converts the 195+133 to 197, and therefore ends up with the same
> filename.
>
> Our UTF-8 conversion routines are based upon the UTF-8 standard, and
> are designed to the best of our ability to work with multiple
> languages and encodings, including European and Asian languages.
> Unfortunately, given a random string of bytes, there is no way to
> definitively determine that this is a UTF-8 versus native string.
> Hopefully, the case given by Heinz will occur rarely if ever.
>
> Cheers,
> Chris
> ITTVIS
Thank you very much for this explanation.
I share your view, that this "pathological case" under normal
circumstances will occur rarely if ever. On the other hand it is a
very serious thing to delete wrong files or even wrong directories. I
wonder if it is the concept of UTF-8 to convert strings to extended
ASCII - as far as possible. In this way we get a mix of filenames,
some encoded in extended ASCII and others in UTF-8.
I am afraid that this approach will make trouble for IDL users again
and again. Aren't there better solutions?
Would it be possible to use UTF-8 filenames for the I/O routines in
principle? Or at least if a corresponding keyword is set? I myself
felt fine with the IDL 7 approach.
You could add the conversion routines separately to the builtin
functions. This would give users the option to do the conversion if
wanted. This furthermore would not break old code.
Regards, Heinz
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
