[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

bug#30520: sed: silently fail substitution for 2GB+ single-line file

From: YushiOMOTE
Subject: bug#30520: sed: silently fail substitution for 2GB+ single-line file
Date: Mon, 19 Feb 2018 06:51:04 +0000

The following behavior is observed with sed version 4.2.2 (x86_64 / CentOS

(It's also observed with the current head of the git repo:

When I pass a 2GB+-single-line to sed for substitution, sed exits with
success status code but it silently fails substitution.

$ sed `s/aaa/bbb/g` 20GB+-single-line-file.txt > out.txt && echo OK
(...but nothing gets substituted even though the target string exists)

In summary, the cause seems that sed isn't checking error return values
from address@hidden

What is happening in detail seems as follows.

At the beginning of `do_subst` function (sed/execute.c),

  /* The first part of the loop optimizes s/xxx// when xxx is at the
     start, and s/xxx$// */
  if (!match_regex(sub->regx, line.active, line.length, start,
                   &regs, sub->max_id + 1))

`line.length` (unsigned long), is set to the value larger than INT_MAX. And
when `match_regex` (sed/regexp.c) calls `re_search`,

    ret = re_search (&regex->pattern, buf, buflen, buf_start_offset,
                     buflen - buf_start_offset,
                     regsize ? regarray : NULL);

`buflen` here is equal to `line.length` (the value larger than INT_MAX) but
the corresponding argument of `re_search` is `int`. The value is converted
to a negative number. Then, `re_search` returns -1 due to its error
checking. But the return value is not treated by sed and no errors are

A possible fix could be adding an error checking for `re_search` (like this
Or, we can check if the value of line length exceeds INT_MAX before
reaching `re_search`. (`grep` checks it and returns with error status code)

Best regards,


Attachment: 0001-sed-check-errors-of-re_search.patch
Description: Source code patch

reply via email to

[Prev in Thread] Current Thread [Next in Thread]