Re: [Nano-devel] [PATCH] syntax: lua: fix word boundaries on standard library functions

[Tom Levy]

Hi Benno,

Thanks for applying my patch!


1.

>>   # File handle methods
>>   color brightyellow "\:\<(close|flush|lines|read|seek|setvbuf|write)\>"
>
> In this regex, I think the leading backslash is unneeded, and the \< is
> unneeded too.  Can you confirm?  (As I have no clue about Lua.)

Confirmed (disclaimer: I am not a Lua expert). There is a similar
unneeded \> in the "Keywords" rule (below). I'd fix the \: but keep
the \< for symmetry. What do you think?

2.

> # Keywords
> color brightyellow "\<(io|math|os|string|table|coroutine|debug)\>\."
>
> Isn't this superfluous?  Because the regexes that you corrected will
> color those same strings again -- at least, if the used functions are
> covered by those regexes.  Do "your" regexes cover *all* existing
> functions, or could there be more?

The "Standard library" rules (the ones I corrected) are missing some
functions (e.g. math.sin, math.sqrt, table.unpack). I am working on a
patch.

I don't know why the above regex was originally added. If we only had
to highlight valid code and the "Standard library" rules were always
up-to-date, I would remove it. As it is, I can think of some reasons
to keep it:

a. If I write invalid code such as "io.foo(...)", nano will highlight
"io." but not "foo". This correctly tells me that "io" is a known
library but it has no function named "foo". It looks odd (I would
expect only "io" to be highlighted), but that might be considered a
good thing because it draws attention to a potential error.

b. Robustness. The function "table.unpack" is missing from the
"Standard library" rule but "table.unpack(...)" still gets highlighted
correctly because "table." matches the above regex and "unpack("
matches another regex from the "Keywords" section. This is not a
coincidence: "unpack" used to be a global function but was moved to
the "table" library in Lua 5.2. If other functions are moved this way
in future, the above regex would handle them. On the flip side, this
robustness works against point (a) because invalid code such as
"math.unpack(...)" gets highlighted as if it was valid.

Personally, I don't mind if the regex is kept or removed.

3.

> # Hex literals
> color red "0x[0-9a-fA-F]*"
>
> The * should be +, right?  Because 0x by itself would be wrong?

Correct. The description in the reference manual is not precise, but
the Lua interpreter rejects 0x so it's definitely illegal (tested with
Lua 5.3). This regex should also use \< and \>.


Some new things:

4.

> # Numbers
> color red "\<([0-9]+)\>"
>
> # Symbols
> color brightmagenta "(\(|\)|\[|\]|\{|\})"

The outer parentheses in both regexes are redundant.

5.

> # Numbers
> color red "\<([0-9]+)\>"

Numbers can have a decimal point and an exponent (e.g. 1.2e3 == 1200).
Currently a decimal point with no exponent looks fine (digits are red,
point is black). However, exponents look horrible (the "e" and the
digits on either side are not highlighted). I would do it like this:

icolor red "\<[0-9]+(\.[0-9]*)?(e[+-]?[0-9]+)?\>"
icolor red "\<\.[0-9]+(e[+-]?[0-9]+)?\>"

These rules are good, but not perfect: "print(1.)" is valid Lua, but
the "." would not be highlighted because \> only matches after a word
character. The "1" would be highlighted though, which I think is good
enough.

6.

> # Hex literals
> color red "0x[0-9a-fA-F]*"

Hex literals got more complicated in Lua 5.2:

> Hexadecimal constants also accept an optional fractional part plus
> an optional binary exponent, marked by a letter 'p' or 'P'.

(from https://www.lua.org/manual/5.2/manual.html#3.1 , search for
"numerical constant").

I would do something analogous to the decimal rules above:

icolor red "\<0x[0-9a-f]+(\.[0-9a-f]*)?(p[+-]?[0-9]+)?\>"
icolor red "\<0x\.[0-9a-f]+(p[+-]?[0-9]+)?\>"

I used icolor because the 0x is case insensitive since Lua 5.2.

I only found out about hex fractions and exponents when I checked if
0x is valid, and wouldn't mind if this was left out.

7.

> # Strings
> color red "\"(\\.|[^\\\"])*\"|'(\\.|[^\\'])*'"

There are two problems here. Firstly, backslashes inside bracket
expressions "[...]" lose their special meaning so they don't need to
be doubled. Secondly, from doc/nanorc.5:

> Quotes inside these string parameters don't have to be escaped with
> backslashes. The last double quote in the string will be treated as
> its end.

So \" should be changed to ". Result:

color red ""(\\.|[^"\])*"|'(\\.|[^'\])*'"

I also reordered the backslash and double/single quotes in the bracket
expressions to make it clear that they are not escape sequences.

Surprisingly, the original regular expression works correctly despite
the errors.


I'm working on patches for 3-7, waiting for reply regarding 1 and 2.

Tom