bug#24162: 25.0.50; utf8 filenames with ornamented characters in MS-Windows

[cschr]

Okay - thanks so much, Eli !

"M-: (w32-application-type (executable-find "touch")) RET"       evaluates
to "msys"

The files which cannot be processed are normal files created by me locally -
they only thing special about them is the ornamented character in the file
name. They can be processed normally by Windows Explorer, which also shows
the ornamented characters properly.

One last question please, Eli: how do I recognize native windows ports, and
where do I get native ports of all the utilities needed by emacs (ls, diff,
touch, ...)

Thank you very much for your great help !

Regards
Chris

-----Original Message-----
From: Eli Zaretskii [mailto:eliz@gnu.org] 
Sent: Samstag, 6. August 2016 13:53
To: cschr
Cc: 24162@debbugs.gnu.org
Subject: Re: bug#24162: 25.0.50; utf8 filenames with ornamented characters
in MS-Windows

[Please keep the bug address on the CC list.]

> From: "cschr" <cschr@freenet.de>
> Date: Sat, 6 Aug 2016 13:30:18 +0200
> 
> Honestly I am not an emacs expert yet - it could well be that my language
> configuration is not as it should be.   Maybe you could give me a hint how
> to set it, or direct me to the right web page ?

You shouldn't need to do anything, the defaults should "just work" for you.
They do work for me.  However, using MSYS2 or Cygwin builds of various
utilities might be a problem, and could very well cause incompatibilities.
The only way around that is to use native ports of those utilities.

> Im indeed using the MSYS2 build of the GNU touch utility.

That doesn't tell enough about the nature of the binary.  What does the
following command display inside Emacs?

  M-: (w32-application-type (executable-find "touch")) RET

> But the strange thing is: I also downloaded and used emacs via MSYS2
> - and it had the same problem !

Emacs downloaded from MSYS2 is a native Windows program.  I'm not so sure
about touch.exe; the above command should tell.

> I had had an earlier issue with ornamented characters which was even
worse:
> dired sometimes displayed files with ornamented characters incorrectly 
> and was unable to open them (but this did not happen always - I never 
> found out when it did and when not!).

Could it be that the files that cannot be displayed were created by some
external utilities?  Or by old versions of Emacs?  What does the Windows
Explorer show when you try to display the folders with those file names?

> I had resolved this one successfully by putting the following into my 
> init.el - since then the issue never occurred again:
>  (prefer-coding-system 'utf-8)
> (set-default-coding-systems 'utf-8)
> (set-terminal-coding-system 'utf-8)
> (set-keyboard-coding-system 'utf-8)
> ;; backwards compatibility as default-buffer-file-coding-system ;; is 
> deprecated in 23.2.
> (if (boundp 'buffer-file-coding-system)
>     (setq-default buffer-file-coding-system 'utf-8)
>   (setq default-buffer-file-coding-system 'utf-8)) ;; Treat clipboard 
> input as UTF-8 string first; compound text next, etc.
> (setq x-select-request-type '(UTF8_STRING COMPOUND_TEXT TEXT STRING))

These are all wrong, you should remove them from your init file.  The
correct defaults for the various coding-systems are set up on Windows
automatically, and you shouldn't try changing them, as long as you are using
the native Windows build of Emacs.

> what I now just tried instead is: 
> 1. I took out all "...-coding-system" statements from my init.el and 
> restarted emacs 2. I started dired and looked at the folder "täst.txt" 
> - result: now the ornamented character is displayed uncorrectly and 
> the file cannot be opened anymore

Can you display that folder in the Windows Explorer?  If so, does it show
correct file names?  If the file names are corrupted in the Explorer, you
have somehow created that folder with file names in the wrong encoding.

> 3. I executed again > 
> C:\Programme\Emacs-24.5-Win64\bin\emacsclientw.exe -n -a
C:\Programme\Emacs-24.5-Win64\bin\runemacs.exe -c "täst.txt" - result:
> now this file opened properly

As expected.

> I now also did another test with a minimum init.el which just had 
> "(server-start)" - everything worked fine - but this does not mean 
> that the dired-problem is not there, because this one only occurs 
> sometimes

As long as you use native Windows programs to create files, and as long as
your Emacs doesn't have those wrong settings of coding systems, you should
not see such problems.  At least not in Emacs 24.5 and later.

> So is this a config problem, and if yes:  can you give me a hint how 
> to get the right setup in order to cover all 3 problems ?

The advice is (a) don't change any of the coding-systems in your init file,
and (b) use only native Windows utilities outside of Emacs to create/modify
files whose names include non-ASCII characters.