bug#32806: An assertion is hit when processing "(|()()0)\2" extended regular expression

[Anatoly Trosinenko]

Hello,

When processing a simple extended regular expression, GNU grep hits an
assertion. This is reproducible on the latest commit from master
branch 51dacfb (Sep 19).

$ export LC_ALL=C
$ gdb -q --args ./grep -E "(|()()0)\2"
Reading symbols from ./grep...(no debugging symbols found)...done.
(gdb) r
Starting program: /path/to/grep -E \(\|\(\)\(\)0\)\\2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
grep: regexec.c:1342: pop_fail_stack: Assertion `num >= 0' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (address@hidden) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (address@hidden) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff75ae801 in __GI_abort () at abort.c:79
#2  0x00007ffff759e39a in __assert_fail_base (fmt=0x7ffff77257d8
"%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
address@hidden "num >= 0",
address@hidden "regexec.c", address@hidden,
    address@hidden <__PRETTY_FUNCTION__.12590>
"pop_fail_stack") at assert.c:92
#3  0x00007ffff759e412 in __GI___assert_fail
(address@hidden "num >= 0",
address@hidden "regexec.c", address@hidden,
address@hidden <__PRETTY_FUNCTION__.12590>
"pop_fail_stack") at assert.c:101
#4  0x00007ffff758f3f8 in pop_fail_stack
(eps_via_nodes=0x7fffffffd5d0, regs=0x55555578b110, nregs=4,
pidx=<optimized out>, fs=<optimized out>) at regexec.c:1342
#5  0x00007ffff76650e8 in pop_fail_stack (eps_via_nodes=<optimized
out>, regs=<optimized out>, nregs=<optimized out>, pidx=<optimized
out>, fs=<optimized out>) at regexec.c:1427
#6  set_regs (address@hidden,
address@hidden, address@hidden,
address@hidden, fl_backtrack=<optimized out>) at
regexec.c:1419
#7  0x00007ffff766e272 in re_search_internal
(address@hidden, address@hidden
"\n", address@hidden, start=<optimized out>, address@hidden,
range=<optimized out>, address@hidden, address@hidden, nmatch=4,
pmatch=0x55555578b110,
    eflags=0) at regexec.c:846
#8  0x00007ffff766eb88 in re_search_stub (bufp=0x555555788cd0,
string=<optimized out>, length=<optimized out>, start=0,
range=<optimized out>, stop=<optimized out>, regs=0x5555557883a0,
ret_len=<optimized out>) at regexec.c:420
#9  0x00007ffff766f7e0 in __re_search (bufp=<optimized out>,
string=<optimized out>, length=<optimized out>, start=<optimized out>,
range=<optimized out>, regs=<optimized out>) at regexec.c:291
#10 0x0000555555558857 in EGexecute ()
#11 0x000055555555df08 in main ()


Best regards
Anatoly