Re: [Bug-zile] Adding Unicode support

[Gary V. Vaughan]

Hi Garnaik,

On Mar 13, 2014, at 9:11 AM, Garnaik Sumeet <address@hidden> wrote:

> Hello Gary and Reuben Sir,
>          I kind of overviewed the source of zile and hopefully I have figured 
> it out on how to add unicode support to zile.

Excellent!

> I believe the following files in the zile library will get really few 
> changes,the code is really well structured and not much of it will get 
> affected to add support for UTF-8.
> 
> /zile/bind.lua
> /zile/buffer.lua
> /zile/completion.lua
> /zile/file.lua
> /zile/keycode.lua
> /zile/search.lua
> /zile/term_curses.lua
> /zile/term_minibuf.lua
> /zile/term_redisplay.lua
> /zile/window.lua
> /zmacs/specs/spec_helper.lua
> /zz/eval.lua

Perhaps not even all of those, though I didn't look in detail myself yet, so 
there're probably still some corner cases I haven't considered so far, which 
you spotted in your review.

Lua is already 8-bit clean, so as long as all the places that manipulate buffer 
text use utf8 character offsets instead of byte offsets that takes care of 
displaying unicode correctly.  I haven't thought too much about utf8 character 
entry, but that code is reasonably self-contained too, so should be pretty 
straight forward.  As far as zmacs is concerned, it must follow Emacs' example.

Note that across Zile (and zmacs in particular) we're currently transitioning 
from a hand-rolled test harness that is executed with 'make tests-check-local' 
to a Specl framework that has only just gained the Zmacs infrastructure it 
needed to operate.  Until that transition is complete, you should check that 
'make tests-check-local' continues to work after each of your changes.  New 
features should come with specl tests that verify their behavior, though it's 
not always straight forward to figure out a good way to test things that rely 
on the curses display, so please ask for help if you get stuck.  I find writing 
the tests first really does help, because you can be sure that they fail when 
the feature is broken/not-yet-implemented, and the act of writing comprehensive 
tests helps to crystallize one's design before tackling the implementation 
code.  Also, add new tests as you think of them during the implementation - or 
at least empty test descriptions to remind you to go back and flesh out the 
details later.  I recommend you work the same way - Specl is quite well 
documented, comes with literally hundreds of examples, and is pretty straight 
forward to use: And I'm happy to hear of any bugs or usability issues you have 
with it, so that I can fix them promptly.

Talking of testing: I'm in the design stage of adding some virtual terminal 
support to Specl, so Specl 12 will be able to run curses programs -like zmacs- 
in an internal VT, and then verify that selected rectangles in the display 
array of char+attributes cells have the expected contents.  But first, Specl 11 
will be out in a few days I think, and already has many improvements over the 
current release.  Immediately after Specl 11, I'll turn my attention back to 
migrating the Zile tests to the new specl framework.

> There a few libraries that provide support for 
> unicode.....slnunicode(available in the luarocks repository)......but I also 
> found another one
> 
> https://github.com/starwing/luautf8

These are the same two I have.  There are some small problems with each one, so 
it might pay to spend a little time trying to do some of the basic integration 
with both to get a feel for which is the better fit for Zile.  If you like 
slunicode, then it will need some patches to add support for Lua 5.2 - Zile is 
currently Lua 5.2 only (though I might try and support 5.1 again in the near 
future) and slunicode is currently Lua 5.1 only; on the other hand 
starwing/luautf8 seems to work in 5.2 already, but has no LuaRock package, so 
if you choose that one, we should try to write a rockspec for it and have it 
added to the luarocks repository.

> Since the data structure in zile are mutable and lua performs all the 
> operations in terms of bytes.....the only thing we need to care about in case 
> of utf-8 is for functions that manipulate the strings and functions that 
> display the chars on the screen(lcurses takes care of these since it uses 
> alien to give a complete binding to the ncurses library).

While it's true that we use alien to manipulate the buffer text bytes, the 
display of buffers in the terminal is handled by lcurses (part of luaposix).  
It's possible that we'll need to ensure lcurses is linked the wide-character 
version of libncurses to support proper display of unicode, but we can figure 
that out once we can handle utf8 file contents.

> Hopefully I have got a correct idea of the code base of zile.Eagerly awaiting 
> for you feedback.

It seems as though you are already moving firmly in the right direction.  Look 
forward to seeing your further progress.

Cheers,
-- 
Gary V. Vaughan (gary AT vaughan DOT pe)

signature.asc
Description: Message signed with OpenPGP using GPGMail