bug#18580: [PATCH] dfa: check end of an input buffer after a transition in non-UTF8 multibyte locales

[Jim Meyering]

On Sat, Oct 4, 2014 at 9:37 AM, Paul Eggert <address@hidden> wrote:
> Norihiro Tanaka wrote:
>>
>> However, I also confirmed an additional byte is required at the
>> end of the input buffer.  dfaexec will temporarily replace it with
>> eolbyte as sentinel.
>
>
> Thanks, I pushed that after adjusting the checkin log message.  I will try
> to get to the other patches in this bug report soon.

Actually, we need yet another byte at the end, and one more prior:

When I built with ASAN and rawhide's gcc version 4.9.1 20140930
(Red Hat 4.9.1-11) (GCC), using this command:

  make CFLAGS=-ggdb3 AM_CFLAGS=-fsanitize=address \
    AM_LDFLAGS='-fsanitize=address -static-libasan' check

I saw two test failures. You can see that the first test triggers an access
one past the end, and all others trigger an access one prior to the beginning.
Here is a summary of the problems:

  $ grep Memory tests/inconsistent-range.log tests/empty.log
  tests/inconsistent-range.log:    [32, 34) 'eolbytes' <== Memory
access at offset 34 overflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable
  tests/empty.log:    [32, 34) 'eolbytes' <== Memory access at offset
31 underflows this variable

Here are the first two, in more detail:

  ==25556==ERROR: AddressSanitizer: stack-buffer-overflow on address
0x7fff1d5f8432 at pc 0x0000004b63e5 bp 0x7fff1d5f7e20 sp
0x7fff1d5f7e18
  READ of size 1 at 0x7fff1d5f8432 thread T0
      #0 0x4b63e4 in mbs_to_wchar /home/j/w/co/grep/src/dfa.c:482
      #1 0x4c5c81 in transit_state /home/j/w/co/grep/src/dfa.c:3184
      #2 0x4c6d03 in dfaexec_main /home/j/w/co/grep/src/dfa.c:3353
      #3 0x4c782c in dfaexec_mb /home/j/w/co/grep/src/dfa.c:3449
      #4 0x4c78ea in dfaexec /home/j/w/co/grep/src/dfa.c:3466
      #5 0x4ce116 in EGexecute /home/j/w/co/grep/src/dfasearch.c:310
      #6 0x4b4ac9 in main /home/j/w/co/grep/src/grep.c:2518
      #7 0x7f97de4c90df in __libc_start_main (/lib64/libc.so.6+0x200df)
      #8 0x406dd6 (/home/j/w/co/grep/src/grep+0x406dd6)

  Address 0x7fff1d5f8432 is located in stack of thread T0 at offset 34 in frame
      #0 0x4b32fe in main /home/j/w/co/grep/src/grep.c:2099

    This frame has 6 object(s):
      [32, 34) 'eolbytes' <== Memory access at offset 34 overflows this variable
      [96, 104) 'keyalloc'


  ==25501==ERROR: AddressSanitizer: stack-buffer-underflow on address
0x7fff1faadb4f at pc 0x0000004d3a87 bp 0x7fff1faad6e0 sp
0x7fff1faad6d8
  READ of size 1 at 0x7fff1faadb4f thread T0
      #0 0x4d3a86 in bm_delta2_search /home/j/w/co/grep/src/kwset.c:534
      #1 0x4d4e3c in bmexec_trans /home/j/w/co/grep/src/kwset.c:663
      #2 0x4d4f49 in bmexec /home/j/w/co/grep/src/kwset.c:678
      #3 0x4d5d9e in kwsexec /home/j/w/co/grep/src/kwset.c:848
      #4 0x4d691d in Fexecute /home/j/w/co/grep/src/kwsearch.c:128
      #5 0x4b4ac9 in main /home/j/w/co/grep/src/grep.c:2518
      #6 0x7f40210110df in __libc_start_main (/lib64/libc.so.6+0x200df)
      #7 0x406dd6 (/home/j/w/co/grep/src/grep+0x406dd6)

  Address 0x7fff1faadb4f is located in stack of thread T0 at offset 31 in frame
      #0 0x4b32fe in main /home/j/w/co/grep/src/grep.c:2099

    This frame has 6 object(s):
      [32, 34) 'eolbytes' <== Memory access at offset 31 underflows
this variable
      [96, 104) 'keyalloc'

I've attached the patch I am about to push:

0001-grep-avoid-stack-buffer-read-underrun-and-overrun.patch
Description: Binary data