Re: [Bug-zile] Zee anastasis in progress!

[Reuben Thomas]

On 23 October 2012 06:39, Gary V. Vaughan <address@hidden> wrote:
> Hi Reuben,
>
> I'm getting married next week, so my hacking time has been (and until
> after my honeymoon, will continue to be) severely curtailed.

Congratulations, Gary!

I'm sorry, you rather caught me with my pants down: I'm just in the
middle of converting Lua Zile to 5.2, and I have nearly finished (just
have a non-atomic keycode atom bug to fix, I think). When I've done
that I'll make a luarocks rockspec, so it should become much easier to
build. In particular, all the dependencies are now released.

Unfortunately we're rather bearing the brunt of being pioneers in this
area, in several directions: writing standalone apps in Lua, writing
POSIX apps in Lua, and using the GNU autotools with Lua. Following in
the footsteps of the trailblazers, we've been trying to pave roads
through the wilderness. I hope others will actually come and use those
roads, though as I've previously said, I'm taking Zee and leaving! But
I am very keen that I leave Zile well tarmacked, and hence I do
encourage you to come back and have a drive around once I've finished,
which should be before the end of your honeymoon.

> even though luarocks
> are available, they are not ported to Mac, and only build with a lot of
> manual intervention.

alien, luaposix, lrexlib and stdlib should all just build on Mac, as
the rockspecs either just use the underlying build system (alien &
luaposix), which I thought we'd already tested, or the rockspec uses
luarocks's built-in build system, which should work (lrexlib and
stdlib). So this sounds like bugs that should be fixed in the relevant
places.

>   1. It appears Zee requires lua 5.2, but Homebrew doesn't seem to support
>      side-by-side 5.1 and 5.2.  Which is normally fine (you can switch
>      back-and-forth between different versions without reinstalling), except
>      that if you have a luarocks installed over the inactive lua, it refuses
>      to run.

Yes, I'm moving to 5.2, because I can't see any point in staying with
5.1, and I now require __len metamethods. I see this is somewhat
contrary to the general current of the Lua world, but since keeping
libraries compatible with both is not a problem, and nor is parallel
installations (though luarocks needs some more help to make it easy),
I don't propose to change that.

>      Looks like I need to build my own 5.1/5.2 capable dev environment, each
>      with it's own luarocks installation unless I can figure out how to get
>      luarocks built on 5.2 to install rocks to the lua 5.1 paths...

You just need two luarocks installations. I think that's fine, just
think of it as being like having lua5.1 and lua5.2: you also have
luarocks5.1 and luarocks5.2.

I have just deleted my Lua 5.1 environment.

>   2. lrex-gnu picks up Apple's BSD regex.h, and ends up with unresolved
>      symbols.  I managed to download the latest release zip, and fudge the
>      luarocks build spec to link against manually compiled gnu regex.[ch]
>      and supporting files from gnulib to get a working lrex_gnu.so.

A recipe for that would obviously help for non-GNU systems. lrexlib
basically assumes you have glibc.

>      I don't think the luarocks build specification is advanced enough to
>      be patched so that it will ship, compile and link against those extra
>      objects where GNU regex is not standard on the build machine.

Ship source in-tree?

>      I'm also worried that without more compiler specific help (libtool?),
>      the gnu regex symbols will leak out of lrex_gnu.so and clash with any
>      client that wants to use it, but is also linked against it's own gnu
>      regex objects from gnulib (or elsewhere).

The obvious solution is a GNU autotools build system for lrexlib. I
have met with resistance to this from the Windows-using maintainer of
lrexlib in the past; that's why I instead replaced the original
Makefile-based build system with luarocks (I thought this would be a
neat way to be able to remove the parallel mingwin build system, also
Make-based). However, it turned out Shmuel didn't have luarocks and
doesn't want to install it, so actually it's no advance in that sense
(though it is simpler and more functional than the original system).
Hence, an autotools-based system with rockspec to drive it à la
luaposix would be fine. However, I'm not going to write it!

I've analyzed the GNU regex code to see if I can stop using it in Zee.
In particular, re_search_2 is actually useless, since internally it
copies the two strings passed and splices them together. Hence, Zee
now avoids searching from anywhere other than the current cursor
position. On the other hand, backwards searching (the other feature we
need) *is* efficient in rexlib, and I don't think any of the other
libraries that are bound by lrexlib support it.

>   3. alien requires a libffi that's newer than the one shipped in the
>      standard locations by Apple.

You should be able to point luarocks at this with an incantation like:

luarocks5.2 make alien-0.7.0-1.rockspec FFI_LIBDIR=/usr/lib/x86_64-linux-gnu

This is what I used to do before fixing luarocks to know about all the
standard system library locations.

Again, if anything you learn can be added to the luarocks.org wiki
and/or submitted in patch form to luarocks and/or brew, that would
obviously make things better in future. I'm encouraging the Debian
maintainers to
add some of my multi-Lua-version-fu to the Debian version of luarocks;
for now I've held off on also trying to push my multiarch-fu. But yes,
this stuff all needs pushing upstream/into distros to make it
convenient for all users.

>   4. With all the hand compiled modules in place, after './bootstrap &&
>      ./configure' in zee HEAD, 'make' fails when help2man tries to run
>      'src/zee --help' because Mac OS doesn't have memrchr.
>
>      I see you have a TODO to write a fallback... where would that live?
>      memrchr.c -> memrchr.so in zee?  An enhancement to alien?  Maybe in
>      luaposix.so?  There is a nice implementation of memrchr() in gnulib
>      already, but I don't know where to put the object for best separation
>      of concerns and transparent calling with alien from Zee.

luaposix I've always tried to keep purely POSIX, and it's BSD
licensed. Adding C to Zee would require the reintroduction of a C
build system. What I'd prefer is C bindings for libgnu(lib). Sadly, I
don't think libgnulib exists, and I presume memrchr isn't in libposix.
Basically, this is more bureaucracy than I care to think about right
now. Another luxury I intend to permit myself with Zee is to develop
just for GNU, and leave other people to port it if they like. I have
spent a long time worrying about portability, and I'd like to program
along another dimension for a bit.

> Next up, Lua Zile:
>
>   1. Lua Zile still wants lua-5.1 and alien-0.6.1 according to loadlua.lua.in.

I've not yet committed the relevant changes. I've removed ldb, because
I never used it; I don't think it should be too much code to fix it
up; and maybe someone already has. The death of luaforge is a real
backwards step, as there's no CPAN for Lua, so lots of people just
have their own private versions of stuff; at least many of them are on
github.

Hence, the rest of the work you did on getting things working with 5.2
should be unnecessary (sorry).

I agree with your summary: probably the best thing you can work on is
the necessary improvements to luarocks/the lrexlib build system to get
lrex_gnu to build on non-GNU systems, as that's the one essential big
thing I'm not going to tackle.

-- 
http://rrt.sc3d.org