Re: Dealing with character ranges in grep

[Paolo Bonzini]

[making this public, there should be no reason not to]

On 06/08/2011 10:14 PM, Aharon Robbins wrote:
> Hi.  As we've discussed a little previously, I finally got tired of
> trying to explain to users why the character range [a-z] was matching
> most uppercase letters also.  ("I've found a bug in gawk! [a-z] matches 'C' !"
> "No - it's a POSIX locale issue".)  This had to be the most F of the FAQs.
>
> So, for the upcoming gawk 4.0, I decided (as Karl put it) to cut the
> Gordian knot and make ranges behave like the C locale, the way it's long
> been documented, and as most people expect.  Those who want the POSIX
> behavior can still get it using --posix.
>
> So I went back and just made the fix in the dfa and regex code, by
> introducing a new syntax bit, RE_RANGES_IGNORE_LOCALES, which is turned
> on in RE_SYNTAX_GNU_AWK and added to RE_SYNTAX_AWK for gawk's --traditional
> option.  (This turned out to be easier than I'd feared it would be.)

Actually, it should be even easier, for two reasons.

First reason: using wcscoll is quite broken, even more so than collation 
equivalent ordering.  Besides, we should be in control of the non-_LIBC 
cases, so we should submit a patch to glibc (or patch gnulib locally) 
that makes your RE_RANGES_IGNORE_LOCALES the sole possibility when _LIBC 
is not defined.

Second reason: nowadays, dfa.c always punts on parsing of multibyte 
bracketed expressions and defers to regex.  The code that handles 
mbcsets is there in case someone is using dfaexec with a NULL backref 
argument, but we might as well remove it and I wouldn't complain at all. 
 So, dfa.c also does not need any special casing of 
RE_RANGES_IGNORE_LOCALES.  Instead, hard_LC_COLLATE should be removed as 
a premature optimization.

The important point is to realize that you cannot fix the whole problem: 
--without-included-regex will forever yield glibc's CEO (you cannot help 
that, and if distros choose to use it you will still get bogus bug 
reports), while the default choice of --with-included-regex will give 
wchar_t ordering.  The above solution takes this into account, and 
within this constraint it provides a much cleaner result:

1) no need for POSIXLY_CORRECT (which would be an abuse of 
POSIXLY_CORRECT actually... where's POSIX_ME_HARDER when you need it? ;)

2) as a result of (1), no need to say anything in the documentation 
(and, anything you say would likely be incorrect in the 
--without-included-regex case);

3) no need for extra flags and changes to the regex clients;

4) no need to care about consistency between dfa.c and regex definitions;

5) instant applicability of the solution to all GNU packages just by 
upgrading gnulib or importing a new version of regex.

So, unlike before, you sold me on this, *provided the above plan is 
implemented*.  The difference is that this approach, I think, does not 
cause more headaches than it solves.  Hopefully, it will not provide any 
new headache assuming we can synchronize decently a release of gawk, 
grep and sed!

Would it be too much to ask to hold gawk 4.0 until the above plan is 
realized?  It's strictly about gawk/grep/gnulib; no need to involve 
glibc from the beginning.  Even better, would anyone help with the work 
while I'm on vacation (from Saturday till the 26th of June)?

Paolo