[PATCH 0/3] more pre-lexer revisions

[Ben Pfaff]

These patches are more lexer housekeeping while I get the lexer
reimplementation ready.

Comments welcome!

Ben Pfaff (3):
  lexer: New type enum token_type.
  lexer: Get rid of lex_negative_to_dash().
  lexer: Change the functions for retrieving token strings.

 src/data/identifier.c                      |    4 +-
 src/data/identifier.h                      |   21 ++-
 src/language/command.c                     |   16 +-
 src/language/control/loop.c                |   10 +-
 src/language/control/repeat.c              |   30 ++--
 src/language/data-io/combine-files.c       |   26 +-
 src/language/data-io/data-list.c           |   42 ++--
 src/language/data-io/data-reader.c         |    2 +-
 src/language/data-io/file-handle.q         |   17 +-
 src/language/data-io/get-data.c            |   77 ++++---
 src/language/data-io/get.c                 |   10 +-
 src/language/data-io/inpt-pgm.c            |   12 +-
 src/language/data-io/placement-parser.c    |   38 ++--
 src/language/data-io/print-space.c         |    8 +-
 src/language/data-io/print.c               |   22 +-
 src/language/data-io/save-translate.c      |   30 ++--
 src/language/data-io/save.c                |   16 +-
 src/language/data-io/trim.c                |   22 +-
 src/language/dictionary/apply-dictionary.c |    4 +-
 src/language/dictionary/attributes.c       |   26 +-
 src/language/dictionary/formats.c          |    8 +-
 src/language/dictionary/missing-values.c   |   20 +-
 src/language/dictionary/modify-variables.c |   36 ++--
 src/language/dictionary/mrsets.c           |   39 ++--
 src/language/dictionary/numeric.c          |   12 +-
 src/language/dictionary/rename-variables.c |    8 +-
 src/language/dictionary/sys-file-info.c    |    8 +-
 src/language/dictionary/value-labels.c     |   18 +-
 src/language/dictionary/variable-display.c |   26 +-
 src/language/dictionary/variable-label.c   |    8 +-
 src/language/dictionary/vector.c           |   24 +-
 src/language/expressions/evaluate.c        |   14 +-
 src/language/expressions/parse.c           |   92 +++++----
 src/language/lexer/format-parser.c         |    6 +-
 src/language/lexer/lexer.c                 |  339 +++++++++++++++++++---------
 src/language/lexer/lexer.h                 |   19 +-
 src/language/lexer/q2c.c                   |   36 ++--
 src/language/lexer/value-parser.c          |    9 +-
 src/language/lexer/variable-parser.c       |   18 +-
 src/language/stats/aggregate.c             |   36 ++--
 src/language/stats/autorecode.c            |    8 +-
 src/language/stats/correlations.c          |   26 +-
 src/language/stats/crosstabs.q             |   14 +-
 src/language/stats/descriptives.c          |   46 ++--
 src/language/stats/examine.q               |   24 +-
 src/language/stats/factor.c                |   64 +++---
 src/language/stats/flip.c                  |   10 +-
 src/language/stats/frequencies.q           |   21 +-
 src/language/stats/glm.c                   |   14 +-
 src/language/stats/npar.c                  |   88 ++++----
 src/language/stats/oneway.c                |   20 +-
 src/language/stats/rank.q                  |   18 +-
 src/language/stats/regression.q            |    6 +-
 src/language/stats/reliability.c           |   28 ++--
 src/language/stats/roc.c                   |   46 ++--
 src/language/stats/sort-cases.c            |    6 +-
 src/language/stats/sort-criteria.c         |    8 +-
 src/language/stats/t-test.q                |   18 +-
 src/language/tests/float-format.c          |   20 +-
 src/language/tests/format-guesser-test.c   |    4 +-
 src/language/tests/moments-test.c          |    6 +-
 src/language/tests/paper-size.c            |    4 +-
 src/language/utilities/cd.c                |    4 +-
 src/language/utilities/echo.c              |    2 +-
 src/language/utilities/host.c              |   10 +-
 src/language/utilities/include.c           |   16 +-
 src/language/utilities/permissions.c       |   10 +-
 src/language/utilities/set.q               |   39 ++--
 src/language/utilities/title.c             |    9 +-
 src/language/xforms/compute.c              |   16 +-
 src/language/xforms/count.c                |   26 +-
 src/language/xforms/recode.c               |   34 ++--
 src/language/xforms/select-if.c            |    6 +-
 tests/language/expressions/evaluate.at     |    2 +-
 74 files changed, 1021 insertions(+), 861 deletions(-)