[Bison-Announce] Bison 3.7.5 released

[Akim Demaille]

This release fixes several issues found in Bison 3.7.4.  Many thanks to the
reporters (Albert Chin, Balázs Scheidler, Joe Nelson, Jot Dot, Martin Rehak,
Michal Bartkowiak, Zartaj Majeed) and to Vincent Imbimbo for addressing the
issues in counterexample generation.

Bison 3.7 introduced the generation of counterexamples for conflicts,
contributed by Vincent Imbimbo.  For instance on a grammar featuring the
infamous "dangling else" problem, "bison -Wcounterexamples" now gives:

    $ bison -Wcounterexamples else.y
    else.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
    else.y: warning: shift/reduce conflict on token "else" [-Wcounterexamples]
      Example: "if" exp "then" "if" exp "then" exp • "else" exp
      Shift derivation
        exp
        ↳ "if" exp "then" exp
                          ↳ "if" exp "then" exp • "else" exp
      Reduce derivation
        exp
        ↳ "if" exp "then" exp                     "else" exp
                          ↳ "if" exp "then" exp •

which actually proves that the grammar is ambiguous by exhibiting a text
sample with two parse trees.

Cheers!

==================================================================

Here are the compressed sources:
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.gz   (5.1MB)
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.lz   (3.1MB)
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.xz   (3.1MB)

Here are the GPG detached signatures[*]:
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.gz.sig
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.lz.sig
  https://ftp.gnu.org/gnu/bison/bison-3.7.5.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify bison-3.7.5.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 0DDCAA3278D5264E

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.69b.24-e9bfc
  Automake 1.16b
  Flex 2.6.4
  Gettext 0.20.1.153-6c39c
  Gnulib v0.1-3995-g839ed059f

==================================================================

GNU Bison is a general-purpose parser generator that converts an annotated
context-free grammar into a deterministic LR or generalized LR (GLR) parser
employing LALR(1) parser tables.  Bison can also generate IELR(1) or
canonical LR(1) parser tables.  Once you are proficient with Bison, you can
use it to develop a wide range of language parsers, from those used in
simple desk calculators to complex programming languages.

Bison is upward compatible with Yacc: all properly-written Yacc grammars
work with Bison with no change.  Anyone familiar with Yacc should be able to
use Bison with little trouble.  You need to be fluent in C, C++ or Java
programming in order to use Bison.

Bison and the parsers it generates are portable, they do not require any
specific compilers.

GNU Bison's home page is https://gnu.org/software/bison/.

==================================================================

NEWS

* Noteworthy changes in release 3.7.5 (2021-01-24) [stable]

** Bug fixes

*** Counterexample Generation

  In some cases counterexample generation could crash.  This is fixed.

*** Fix Table Generation

  In some very rare conditions, when there are many useless tokens, it was
  possible to generate incorrect parsers.

*** GLR parsers now support %merge together with api.value.type=union.

*** C++ parsers use noexcept in more places.

*** Generated parsers avoid some warnings about signedness issues.

*** C-language parsers now avoid warnings from pedantic clang.

*** C-language parsers now work around quirks of HP-UX 11.23 (2003).


* Noteworthy changes in release 3.7.4 (2020-11-14) [stable]

** Bug fixes

*** Bug fixes in yacc.c

  In Yacc mode, all the tokens are defined twice: once as an enum, and then
  as a macro.  YYEMPTY was missing its macro.

*** Bug fixes in lalr1.cc

  The lalr1.cc skeleton used to emit internal assertions (using YY_ASSERT)
  even when the `parse.assert` %define variable is not enabled.  It no
  longer does.

  The private internal macro YY_ASSERT now obeys the `api.prefix` %define
  variable.

  When there is a very large number of tokens, some assertions could be long
  enough to hit arbitrary limits in Visual C++.  They have been rewritten to
  work around this limitation.

** Changes

  The YYBISON macro in generated "regular C parsers" (from the "yacc.c"
  skeleton) used to be defined to 1.  It is now defined to the version of
  Bison as an integer (e.g., 30704 for version 3.7.4).


* Noteworthy changes in release 3.7.3 (2020-10-13) [stable]

** Bug fixes

  Fix concurrent build issues.

  The bison executable is no longer linked uselessly against libreadline.

  Fix incorrect use of yytname in glr.cc.


* Noteworthy changes in release 3.7.2 (2020-09-05) [stable]

  This release of Bison fixes all known bugs reported for Bison in MITRE's
  Common Vulnerabilities and Exposures (CVE) system.  These vulnerabilities
  are only about bison-the-program itself, not the generated code.

  Although these bugs are typically irrelevant to how Bison is used, they
  are worth fixing if only to give users peace of mind.

  There is no known vulnerability in the generated parsers.

** Bug fixes

  Fix concurrent build issues (introduced in Bison 3.5).

  Push parsers always use YYMALLOC/YYFREE (no direct calls to malloc/free).

  Fix portability issues of the test suite, and of bison itself.

  Some unlikely crashes found by fuzzing have been fixed.  This is only
  about bison itself, not the generated parsers.


* Noteworthy changes in release 3.7.1 (2020-08-02) [stable]

** Bug fixes

  Crash when a token alias contains a NUL byte.

  Portability issues with libtextstyle.

  Portability issues of Bison itself with MSVC.

** Changes

  Improvements and fixes in the documentation.

  More precise location about symbol type redefinitions.


* Noteworthy changes in release 3.7 (2020-07-23) [stable]

** Deprecated features

  The YYPRINT macro, which works only with yacc.c and only for tokens, was
  obsoleted long ago by %printer, introduced in Bison 1.50 (November 2002).
  It is deprecated and its support will be removed eventually.

  In conformance with the recommendations of the Graphviz team, in the next
  version Bison the option `--graph` will generate a *.gv file by default,
  instead of *.dot.  A transition started in Bison 3.4.

** New features

*** Counterexample Generation

  Contributed by Vincent Imbimbo.

  When given `-Wcounterexamples`/`-Wcex`, bison will now output
  counterexamples for conflicts.

**** Unifying Counterexamples

  Unifying counterexamples are strings which can be parsed in two ways due
  to the conflict.  For example on a grammar that contains the usual
  "dangling else" ambiguity:

    $ bison else.y
    else.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
    else.y: note: rerun with option '-Wcounterexamples' to generate conflict 
counterexamples

    $ bison else.y -Wcex
    else.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
    else.y: warning: shift/reduce conflict on token "else" [-Wcounterexamples]
      Example: "if" exp "then" "if" exp "then" exp • "else" exp
      Shift derivation
        exp
        ↳ "if" exp "then" exp
                          ↳ "if" exp "then" exp • "else" exp
      Example: "if" exp "then" "if" exp "then" exp • "else" exp
      Reduce derivation
        exp
        ↳ "if" exp "then" exp                     "else" exp
                          ↳ "if" exp "then" exp •

  When text styling is enabled, colors are used in the examples and the
  derivations to highlight the structure of both analyses.  In this case,

    "if" exp "then" [ "if" exp "then" exp • ] "else" exp

  vs.

    "if" exp "then" [ "if" exp "then" exp • "else" exp ]


  The counterexamples are "focused", in two different ways.  First, they do
  not clutter the output with all the derivations from the start symbol,
  rather they start on the "conflicted nonterminal". They go straight to the
  point.  Second, they don't "expand" nonterminal symbols uselessly.

**** Nonunifying Counterexamples

  In the case of the dangling else, Bison found an example that can be
  parsed in two ways (therefore proving that the grammar is ambiguous).
  When it cannot find such an example, it instead generates two examples
  that are the same up until the dot:

    $ bison foo.y
    foo.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
    foo.y: note: rerun with option '-Wcounterexamples' to generate conflict 
counterexamples
    foo.y:4.4-7: warning: rule useless in parser due to conflicts [-Wother]
        4 | a: expr
          |    ^~~~

    $ bison -Wcex foo.y
    foo.y: warning: 1 shift/reduce conflict [-Wconflicts-sr]
    foo.y: warning: shift/reduce conflict on token ID [-Wcounterexamples]
      First example: expr • ID ',' ID $end
      Shift derivation
        $accept
        ↳ s                      $end
          ↳ a                 ID
            ↳ expr
              ↳ expr • ID ','
      Second example: expr • ID $end
      Reduce derivation
        $accept
        ↳ s             $end
          ↳ a        ID
            ↳ expr •
    foo.y:4.4-7: warning: rule useless in parser due to conflicts [-Wother]
        4 | a: expr
          |    ^~~~

  In these cases, the parser usually doesn't have enough lookahead to
  differentiate the two given examples.

**** Reports

  Counterexamples are also included in the report when given
  `--report=counterexamples`/`-rcex` (or `--report=all`), with more
  technical details:

    State 7

      1 exp: "if" exp "then" exp •  [$end, "then", "else"]
      2    | "if" exp "then" exp • "else" exp

      "else"  shift, and go to state 8

      "else"    [reduce using rule 1 (exp)]
      $default  reduce using rule 1 (exp)

      shift/reduce conflict on token "else":
          1 exp: "if" exp "then" exp •
          2 exp: "if" exp "then" exp • "else" exp
        Example: "if" exp "then" "if" exp "then" exp • "else" exp
        Shift derivation
          exp
          ↳ "if" exp "then" exp
                            ↳ "if" exp "then" exp • "else" exp
        Example: "if" exp "then" "if" exp "then" exp • "else" exp
        Reduce derivation
          exp
          ↳ "if" exp "then" exp                     "else" exp
                            ↳ "if" exp "then" exp •

*** File prefix mapping

  Contributed by Joshua Watt.

  Bison learned a new argument, `--file-prefix-map OLD=NEW`.  Any file path
  in the output (specifically `#line` directives and `#ifdef` header guards)
  that begins with the prefix OLD will have it replaced with the prefix NEW,
  similar to the `-ffile-prefix-map` in GCC.  This option can be used to
  make bison output reproducible.

** Changes

*** Diagnostics

  When text styling is enabled and the terminal supports it, the warnings
  now include hyperlinks to the documentation.

*** Relocatable installation

  When installed to be relocatable (via `configure --enable-relocatable`),
  bison will now also look for a relocated m4.

*** C++ file names

  The `filename_type` %define variable was renamed `api.filename.type`.
  Instead of

    %define filename_type "symbol"

  write

    %define api.filename.type {symbol}

  (Or let `bison --update` do it for you).

  It now defaults to `const std::string` instead of `std::string`.

*** Deprecated %define variable names

  The following variables have been renamed for consistency.  Backward
  compatibility is ensured, but upgrading is recommended.

    filename_type       -> api.filename.type
    package             -> api.package

*** Push parsers no longer clear their state when parsing is finished

  Previously push-parsers cleared their state when parsing was finished (on
  success and on failure).  This made it impossible to check if there were
  parse errors, since `yynerrs` was also reset.  This can be especially
  troublesome when used in autocompletion, since a parser with error
  recovery would suggest (irrelevant) expected tokens even if there were
  failure.

  Now the parser state can be examined when parsing is finished.  The parser
  state is reset when starting a new parse.

** Documentation

*** Examples

  The bistromathic demonstrates %param and how to quote sources in the error
  messages:

    > 123 456
    1.5-7: syntax error: expected end of file or + or - or * or / or ^ before 
number
        1 | 123 456
          |     ^~~

** Bug fixes

*** Include the generated header (yacc.c)

  Historically, when --defines was used, bison generated a header and pasted
  an exact copy of it into the generated parser implementation file.  Since
  Bison 3.4 it is possible to specify that the header should be `#include`d,
  and how.  For instance

    %define api.header.include {"parse.h"}

  or

    %define api.header.include {<parser/parse.h>}

  Now api.header.include defaults to `"header-basename"`, as was intended in
  Bison 3.4, where `header-basename` is the basename of the generated
  header.  This is disabled when the generated header is `y.tab.h`, to
  comply with Automake's ylwrap.

*** String aliases are faithfully propagated

  Bison used to interpret user strings (i.e., decoding backslash escapes)
  when reading them, and to escape them (i.e., issue non-printable
  characters as backslash escapes, taking the locale into account) when
  outputting them.  As a consequence non-ASCII strings (say in UTF-8) ended
  up "ciphered" as sequences of backslash escapes.  This happened not only
  in the generated sources (where the compiler will reinterpret them), but
  also in all the generated reports (text, xml, html, dot, etc.).  Reports
  were therefore not readable when string aliases were not pure ASCII.
  Worse yet: the output depended on the user's locale.

  Now Bison faithfully treats the string aliases exactly the way the user
  spelled them.  This fixes all the aforementioned problems.  However, now,
  string aliases semantically equivalent but syntactically different (e.g.,
  "A", "\x41", "\101") are considered to be different.

*** Crash when generating IELR

  An old, well hidden, bug in the generation of IELR parsers was fixed.