Re: Japanese '者' (U+8005) is replaced with \350\200

[OKUMURA, Akira]

Dear Karl and Denver,

I am attaching a tar file just in case.

I found that the issue cannot be reproduced on my CentOS 7 virtual machine, on 
which wdiff was installed with yum.

-------------------------
$ uname -a
Linux localhost.localdomain 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ wdiff --version
wdiff (GNU wdiff) 1.2.2

Copyright (C) 1992, 1997, 1998, 1999, 2009, 2010, 2011, 2012 Free Software
Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Written by Franc,ois Pinard <pinard@iro.umontreal.ca>.
-------------------------

But the issue appears only on my macOS 10.15.7, on which wdiff was installed 
via Homebrew. Here is the minimum reproducible example. I guess the root cause 
exists somewhere in macOS libraries (or maybe GNU BSD library difference?) and 
it is probably beyond your maintenance coverage... 

$ cat input1_min.txt
者あ
$ cat input2_min.txt
あ者
$ wdiff input1_min.txt input2_min.txt
[-者あ-]{+あ?+}?

$ uname -a 
Darwin oxon2017.local 19.6.0 Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 
PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64 x86_64
$ otool -L `which wdiff`                        
/usr/local/bin/wdiff:
        /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 
10.0.0, current version 10.6.0)
        
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation 
(compatibility version 150.0.0, current version 1670.10.0)
        /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current 
version 5.4.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current 
version 1281.0.0)

$ wdiff --version
wdiff (GNU wdiff) 1.2.2

Copyright (C) 1992, 1997, 1998, 1999, 2009, 2010, 2011, 2012 Free Software
Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Written by Franc,ois Pinard <pinard@iro.umontreal.ca>.

-- 
OKUMURA, Akira oxon@mac.com / oxon@nagoya-u.jp
⌘ Junior Associate Professor at
- Institute for Space–Earth Environmental Research (ISEE)
- Kobayashi–Maskawa Institute for the Origin of Particles and the Universe (KMI)
Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8601, Japan
Office/Lab/Fax: +81 (0)52-789-4320/4324/4313
skype:okumura.akira

wdiff.tar
Description: Unix tar archive

> On Jan 10, 2021, at 10:53, Denver Gingerich <denver@ossguy.com> wrote:
> 
> On Fri, Jan 08, 2021 at 07:35:19PM -0700, Karl Berry wrote:
>> In general, since the last release of wdiff was in 2014, it would not be
>> surprising if there were bugs in the utf-8 handling in the support code
>> or in wdiff itself.  Denver or Martin, are you there? --best, karl.
> 
> Yes, I'm here.  I don't recall working on anything related to UTF-8 support 
> during my time updating wdiff, nor do I recall specifically testing it.  So I 
> agree with Karl that it would not surprise me if there were bugs in how it 
> handles UTF-8.
> 
> Also, I only saw the two messages from Karl in this thread - please do let me 
> know if I'm missing some other context.  Thanks!
> 
> Denver
> https://jmp.chat/