Re: [Bug-zile] Zee anastasis in progress!

[Gary V. Vaughan]

Hi Reuben,

I'm getting married next week, so my hacking time has been (and until
after my honeymoon, will continue to be) severely curtailed.

In a spare hour last night, I tried to get zee and the newest lua branch
up and running with an eye to rebasing zi to pick up the recent fixes.
However, it is as painful as ever to build everything, even though luarocks
are available, they are not ported to Mac, and only build with a lot of
manual intervention.

Please forgive the brain dump below. This is as much a note to myself so
that I don't forget what I discovered as anything, but advice on how best
to proceed, or better still patches to fix some of the problems I'm stuck
on would be very welcome too :)

First Zee:

  1. It appears Zee requires lua 5.2, but Homebrew doesn't seem to support
     side-by-side 5.1 and 5.2.  Which is normally fine (you can switch
     back-and-forth between different versions without reinstalling), except
     that if you have a luarocks installed over the inactive lua, it refuses
     to run.

     Looks like I need to build my own 5.1/5.2 capable dev environment, each
     with it's own luarocks installation unless I can figure out how to get
     luarocks built on 5.2 to install rocks to the lua 5.1 paths...

  2. lrex-gnu picks up Apple's BSD regex.h, and ends up with unresolved
     symbols.  I managed to download the latest release zip, and fudge the
     luarocks build spec to link against manually compiled gnu regex.[ch]
     and supporting files from gnulib to get a working lrex_gnu.so.

     I don't think the luarocks build specification is advanced enough to
     be patched so that it will ship, compile and link against those extra
     objects where GNU regex is not standard on the build machine.

     I'm also worried that without more compiler specific help (libtool?),
     the gnu regex symbols will leak out of lrex_gnu.so and clash with any
     client that wants to use it, but is also linked against it's own gnu
     regex objects from gnulib (or elsewhere).  The gnulib build tries to
     fix that problem with "#define re_compile rpl_re_compile" etc for each
     entry point in its own config.h, but then lua and/or lrex_gnu.so fails
     to find the undecorated symbols at runtime and bails out with an
     unresolved _re_compile error - for now I've hand compiled everything
     and ignored the potential symbol leak issues, so I can at least
     require "lrex_gnu" without everything falling over :)

  3. alien requires a libffi that's newer than the one shipped in the
     standard locations by Apple.  Homebrew puts a sufficiently new one
     in /usr/local/opt/libffi, but it seems the only way to provide that
     information to the luarocks build is to pull the alien zip file down
     and manually compile and install it with:

       CPPFLAGS=/usr/local/opt/libffi/lib/libffi-3.0.11/include \
       LDFLAGS=/usr/local/opt/libffi/lib luarocks make alien-0.7.0-1.rockspec

     Why doesn't 'luarocks install' respect CPPFLAGS/LDFLAGS?  It looks
     like luarocks config files might be able to help with this if I can
     figure out how to add higher priority library and header search paths
     into it to make it ignore the too-old system libffi.

  4. With all the hand compiled modules in place, after './bootstrap &&
     ./configure' in zee HEAD, 'make' fails when help2man tries to run
     'src/zee --help' because Mac OS doesn't have memrchr.

     I see you have a TODO to write a fallback... where would that live?
     memrchr.c -> memrchr.so in zee?  An enhancement to alien?  Maybe in
     luaposix.so?  There is a nice implementation of memrchr() in gnulib
     already, but I don't know where to put the object for best separation
     of concerns and transparent calling with alien from Zee.

Next up, Lua Zile:

  1. Lua Zile still wants lua-5.1 and alien-0.6.1 according to loadlua.lua.in.

     I made a quick attempt at fixing up the sources to run with lua 5.2
     and the modules I already painstakingly built for the Zee branch, but
     I don't really understand how to idiomatically move from 5.1 setfenv
     to some 5.2 equivalent. I just commented out reliance on that ldb.lua
     just to make progress, and in lisp.lua where I made this naive change:

diff --cc src/lisp.lua                                                          
   
index ffefb1e,ffefb1e..fbf4bf3                                                  
   
--- a/src/lisp.lua                                                              
   
+++ b/src/lisp.lua                                                              
   
@@@ -47,7 -47,7 +47,7 @@@ function Defun (name, argtypes, doc, in               
   
                 arglist = arglist.next                                         
   
                 i = i + 1                                                      
   
               end                                                              
   
--             setfenv (func, setmetatable ({current_prefix_arg = prefix_arg},  
   
++             debug.setupvalue (func, 1, setmetatable ({current_prefix_arg = 
prefix_arg},
                                            {__index = _G, __newindex = _G}))   
   
               prefix_arg = false                                               
   
               local ret = func (unpack (args))                                 
   

     I'm not even sure that is equivalent, and using the debug library in
     the core like that feels wrong too, so I need to wrap my head around
     how _ENV works so that I can rewrite that section of code idiomatically
     when I have everything else working.

  2. At this point, trying to run src/zile falls over in term_redisplay.lua
     with:

Internal error.  Please report this bug with steps to reproduce the problem
/usr/local/share/lua/5.2/base.lua:475: 
stack traceback:
        [C]: in function 'error'
        /usr/local/share/lua/5.2/base.lua:475: in function 'assert'
        .../zile/src/term_redisplay.lua:35: in function 'make_char_printable'
        .../zile/src/term_redisplay.lua:62: in function 'draw_line'
        .../zile/src/term_redisplay.lua:171: in function 'draw_window'
        .../zile/src/term_redisplay.lua:232: in function 'term_redisplay'
        .../zile/src/getkey.lua:57: in function 'getkey'
        .../zile/src/bind.lua:157: in function 'get_key_sequence'
        .../zile/src/bind.lua:80: in function 'get_and_run_command'
        .../zile/src/main.lua:309: in function <.../zile/src/main.lua:218>
        [C]: in function 'xpcall'
        src/zile:49: in main chunk
        [C]: in ?

  3. I need to remove all the Homebrew and manually installed lua builds
     and modules, and work out how to have lua-5.1 with alien-0.6.1 and
     lua-5.2 with alien-0.7.0 side-by-side to confirm that the new lua
     Zile HEAD revision still works with older libraries, and then try
     again with rebasing zi and lpeg syntax-highlighting branches, before
     make an official attempt at porting Lua Zile to Lua 5.2.

I might punt on all of this at my next session, and just try things out
in a Linux VM to confirm that it's all Mac OS portability issues that I
can tackle gradually, before getting drawn into too much low-level work,
and not getting to work on Zile for a while longer as a consequence.

I've also upgraded to latest stdlib and luaposix/curses, which didn't
seem to have any issues, but everything is so broken in my environment
right now, I don't want to rule out any of the changes I made as being
involved in one or more of the issues above.

My current impression is that portability of alien and lrex_gnu in
particular is held back by the luarocks build system, and a few tweaks
to allow passing preferred library and header directories before
calling 'luarocks install' would be a good place to start at getting
things back on track.

lrex_gnu is essentially Linux- (or at least glibc-) only as it stands,
and needs a complete build system overhaul to have any chance of
working out-of-the-box on anything without GNU regex in the system
library.

On 14 Oct 2012, at 07:41, Reuben Thomas <address@hidden> wrote:
> I've done the code review of C Zee. I have a couple of things left to
> integrate, in the terminal display, but the vast bulk is done. I have
> a handful of other TODOs to do.

Would be cool if you could commit your TODO incase I can help out
on anything, or at least not start pulling in the opposite direction
by mistake :)

> As a bonus, I found and fixed one more inefficiency, in reading the
> file: it slurped into a Lua string, then made an AStr from that. It
> now reads directly into an AStr, using alien to bind read. This looks
> like it ought to go in luaposix, but while it's easy to have
> polymorphic arguments in Lua functions (as I have done in lrexlib for
> searching AStrs directly) it's not obviously possible to have
> polymorphic return types without some way of indicating what type you
> want back, which will necessitate changing the interface to
> read/write, e.g. by adding a flag indicating that you'd like a raw
> memory block rather than a string, or perhaps even giving a
> constructor function (more complicated, but more flexible). One to
> think about.

You're several steps ahead of me on this stuff.

However, on a related note, libposix in gnulib has hit a brick wall :(

ASIDE: libposix being a separate library and headers that are a subset
of gnulib to act as a shim between the non-conformance system libc
APIs and the full and correct POSIX APIs, such that with libposix.so
(and headers) installed, everything will just work when your code is
strictly POSIX API only, and you pass -L/usr/local/posix/lib
-I/usr/local/posix/include -lposix to the compiler.

Even so, I think this would be the cleanest solution to some of the
portability concessions made in luaposix, the memrchr dependency in
Zee, and to the requirement for a system GNU regex for lrex_gnu to
build out of the box.  I wonder whether the best way out of the
current mess is to start a minimal libposix project with just enough
in it to support the lua modules we need.  Maybe that would start to
build enough momentum to get an official version through the gnulib
brick wall too.

> The commits are a bunch of big gloopy messes, sorry.

Tsk.  Better than no commits at all though, eh? ;)

> On the other hand it's pretty easy to examine the code to find useful
> patches for Zile/Zi, and even if I'd separated out the commits they
> wouldn't on the whole have applied cleanly: Zee is just too different.

Once I have all of the above working again, I'll start to think about how
to get all of Zile/Zi/Zee into a single branch, and figure out a nice
way to select which flavour you want to play with, but without all
the nasty branch hopping and lua version switching currently entailed!

Cheers,
-- 
Gary V. Vaughan (gary AT gnu DOT org)