[debbugs-tracker] bug#30520: closed (sed: silently fail substitution for 2GB+ single-line file)

[GNU bug Tracking System]

Your message dated Mon, 26 Mar 2018 08:19:33 -0700
with message-id <address@hidden>
and subject line Re: bug#30520: sed: silently fail substitution for 2GB+ 
single-line file
has caused the debbugs.gnu.org bug report #30520,
regarding sed: silently fail substitution for 2GB+ single-line file
to be marked as done.

(If you believe you have received this mail in error, please contact
address@hidden)


-- 
30520: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=30520
GNU Bug Tracking System
Contact address@hidden with problems

--- Begin Message --- Subject: sed: silently fail substitution for 2GB+ single-line file Date: Mon, 19 Feb 2018 06:51:04 +0000

The following behavior is observed with sed version 4.2.2 (x86_64 / CentOS 7.2).

(It's also observed with the current head of the git repo: 41e506e12b363628af29b089f55585b17e1389e6)

When I pass a 2GB+-single-line to sed for substitution, sed exits with success status code but it silently fails substitution.

```

$ sed `s/aaa/bbb/g` 20GB+-single-line-file.txt > out.txt && echo OK

OK

(...but nothing gets substituted even though the target string exists)

```

In summary, the cause seems that sed isn't checking error return values from address@hidden

What is happening in detail seems as follows.

At the beginning of `do_subst` function (sed/execute.c),

```

  /* The first part of the loop optimizes s/xxx// when xxx is at the

     start, and s/xxx$// */

  if (!match_regex(sub->regx, line.active, line.length, start,

                   &regs, sub->max_id + 1))

    return;

```

`line.length` (unsigned long), is set to the value larger than INT_MAX. And when `match_regex` (sed/regexp.c) calls `re_search`,

```

    ret = re_search (&regex->pattern, buf, buflen, buf_start_offset,

                     buflen - buf_start_offset,

                     regsize ? regarray : NULL);

```

`buflen` here is equal to `line.length` (the value larger than INT_MAX) but the corresponding argument of `re_search` is `int`. The value is converted to a negative number. Then, `re_search` returns -1 due to its error checking. But the return value is not treated by sed and no errors are raised.

A possible fix could be adding an error checking for `re_search` (like this patch).

Or, we can check if the value of line length exceeds INT_MAX before reaching `re_search`. (`grep` checks it and returns with error status code)

Best regards,

Yushi

0001-sed-check-errors-of-re_search.patch
Description: Source code patch

--- End Message ---

> --- Begin Message ---
>
>
>
> Subject: 
>
> Re: bug#30520: sed: silently fail substitution for 2GB+ single-line file
>
>
>
>
> Date: 
>
> Mon, 26 Mar 2018 08:19:33 -0700
>
>
>
>
> marking as "done"
>
>
> --- End Message ---