bug#28666: More info regarding binary3 sed test which hangs

[Dennis Clarke]

On 10/04/2017 11:15 AM, Jim Meyering wrote:
> On Sun, Oct 1, 2017 at 7:28 PM, Dennis Clarke <address@hidden> wrote:
> > I looked into the binary3 test a bit and tried this manually :
> >
> > deb8_ppc64$
> > deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp | LC_ALL=C
> > tr -d \\r > binary3.out_manual
> >
> > It just runs endlessly and never returns.
> >
> > Not sure what data to add in here.
>
> Thank you for the details. I presume this is with the latest?

Direct from the release tarball. I also did try a git pull but that
 resulted in other problems.

> Does it also infloop if you remove the output pipe:
>
>    LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out
>
> Can you run it via gdb and hit ^C then see where/why it's looping?
>
>    LC_ALL=C gdb --args ../sed/sed
>
> to run it under gdb, do this:
>
>    (gdb) run -n -f ./binary3.sed < binary.inp

Well, here comes the frustrating problem : it all works now.  :-\

Why?

Your guess is as good as mine but there seems to have been a need for a
lot of locales to be installed as well as an old old dusty configure
option to be removed.   This is really long :

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28665



Assaf Gordon says that "--enable-regex-tests" should not be used.

quote:
    These 'regex' tests are very old, and have not been maintained
    since about 2004. We simply haven't gotten around to either fix
    them or remove them.

Also :

> FAIL: testsuite/utf8-ru.sh

Assaf Gordon says :
    This is one test that will certainly remain even after removing
    the "regex tests".

    As a first step to understand if this is a false positive or
    a sed failure, can you tell us which locales are installed
    on your computer ("locale -a") ?

To this I can tell you that I only had the essentials. A few utf8 and
not many of them.

Assaf Gordon also says :

    I see the test uses "ru_RU.UTF-8", and it's possible it uses the
    locale without first verifying it is available on the system.


I believe that must be true.

After I installed ALL the locale options that I could get as well as
removed the configure option for regex tests we get a clean configure
and a clean compile and testsuite.

So here I am baffled as to what infinite loops were running. I can tell
you that I was running "htop" in an xterm while the system tried and
tried ( vainly ) to perform a gcc bootstrap.  The htop showed me that
there were sed binary tests that had been running for 36 hours plus. I
have done a kill -9 on them two days ago but what result I got from that 
was that the ppid went away to be replaced by pid 1 and the binary tests 
were still running. I was amazed. I ran kill -9 on them again and again 
and was eventually able to make them go away.

Now I can not reproduce them.

I really hate problems like this that seem to be based on some minor
issue with locale or a bad configure option or it is a Tuesday and your
mother's birthday.  Makes no sense whatsoever to try to reproduce now
because it simply runs and run fine.

However let's see what happens anyways :

I will look into my list of attempts over and over for the past 3 days.

deb8_ppc64$ ls -loptr | grep "sed" | grep "^drw" | cut -c31-96
Oct  1 22:55 sed-4.4_linux_3.16.0-4-powerpc64.001/
Oct  1 23:24 sed-4.3_linux_3.16.0-4-powerpc64.001/
Oct  2 11:21 sed-4.4.84-a59fb_linux_3.16.0-4-powerpc64_ppc64.001/
Oct  2 12:06 sed-4.2.2_linux_3.16.0-4-powerpc64.001/
Oct  3 17:27 sed-4.4_linux_3.16.0-4-powerpc64.002/
Oct  3 17:35 sed-4.4_linux_3.16.0-4-powerpc64.002.pkg/
Oct  3 17:40 sed-4.4_linux_3.16.0-4-powerpc64.003/
Oct  3 17:43 sed-4.4_linux_3.16.0-4-powerpc64.003.pkg/

The last two directories that have the suffix ".pkg" are fully clean
and successful builds and tests.  I was so astonished that I did it
twice. Also, installed into /usr/local which is what I really wanted.

Let's go back in time a few days and try the test from within the
first point of failure :

deb8_ppc64$ cd sed-4.4_linux_3.16.0-4-powerpc64.001/testsuite
deb8_ppc64$ LC_ALL=C ../sed/sed -n -f ./binary3.sed < binary.inp > out

gone ... never to return.

deb8_ppc64$ file out
out: empty

It seems to produce nothing.  Let's reach for gdb ( which I also had to
build from sources myself ) :


deb8_ppc64$ LC_ALL=C gdb --args ../sed/sed
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ../sed/sed...done.
(gdb) run -n -f ./binary3.sed < binary.inp
Starting program: 
/usr/local/build/sed-4.4_linux_3.16.0-4-powerpc64.001/sed/sed -n -f 
./binary3.sed < binary.inp

Full 100% CPU usage on a single core and into the infinite depths we go.

Wait a full minute and hit CTRL-C :

^C
Program received signal SIGINT, Interrupt.
match_regex (regex=0x10071b20, buf=0x10040080 
"aa-\nb;9876543210aaaaaaaaaR| D\n\n192\n168\n18", buflen=25, 
buf_start_offset=0,
    regarray=0x0, regsize=0) at sed/regexp.c:356
356       if ((regex->flags & REG_NEWLINE) && buffer_delimiter != '\n')

(gdb) print regex
$1 = (struct regex *) 0x10071b20

(gdb) print *regex
$3 = {pattern = {buffer = 0x10074930, allocated = 224, used = 224, 
syntax = 50790982, fastmap = 0x10071b90 "", translate = 0x0,
    re_nsub = 0, can_be_null = 0, regs_allocated = 1, fastmap_accurate 
= 1, no_sub = 1, not_bol = 0, not_eol = 0,
    newline_anchor = 0}, flags = 0, sz = 2, dfa = 0x10075440, begline = 
false, endline = false, re = "^"}
(gdb)

OKay ... this is at least something to look at. Still makes me wonder
what loop we are in.

Dennis