bug#18266: grep -P and invalid exits with error

[Paul Eggert]

Vincent Lefevre wrote:

>        [...] Note that this option can also be passed to pcre_exec()
>        and pcre_dfa_exec(), to suppress the validity checking of
>        subject strings only. If the same string is being matched
>        many times, the option can be safely set for the second and
>        subsequent matchings to improve performance.
>
> The last sentence would imply that the UTF8 checking is done on the
> whole input buffer before matching is done.

That's pretty subtle, and perhaps too subtle.  A plausible 
interpretation of the phrase "same string is being matched" is that 
libpcre checks only the matched string, and that bytes after the match 
(which did not need to be examined to do the match) are not checked. 
Can you confirm with the libpcre authors that this plausible 
interpretation is incorrect, i.e., that the entire input string is 
checked, even the unmatched part?  If that's what is intended, the 
documentation should state so clearly, so at least there's a 
documentation bug there.

> If there are many invalid UTF8 bytes, this would be slow, IMHO

That's OK.  We don't need grep -P to be fast on invalid input.

> But is the copy of the buffer really needed? Couldn't the invalid
> UTF8 sequences just be replaced by null bytes?

I'd rather not, because that changes the semantics of matching.  The 
null byte is valid input data that might get matched.

> in case of invalid UTF8 bytes, in some (many?) cases, the
> cause is a binary file (possibly with some text in it), where lines
> can be very long. So, wouldn't it mean that it can take significantly
> more memory?

Sure.  But that's the same for -P as it is for plain grep.