First Poke-Conf at Mont-Soleil - A report

[Jose E. Marchesi]

[This is also published in the Applied Pokology blog:
 http://jemarch.net/pokology-20200113.html]

This last weekend we had the first gathering of poke developers, as part
of the GNU Hackers Meeting at Mont-Soleil, in Switzerland.  I can say we
had a lot of fun, and it was a quite productive meeting too: many
patches were written, and many technical aspects designed and clarified.

First we made a little introductory talk for the benefit of the GHM
attendants who were not familiar with poke, followed by a quick review
of the recent developments.  After that, we went to discuss some serious
business: handling of stream-like IO spaces, integral "atoms" in
structs, the adoption of a bug tracking system for the project, how to
best support Unicode and UTF-8 in poke, and many many other things, some
of which are summarized below.

           [Stream-like IO spaces, and stdin/stdout]

How to handle stream-like IO devices (such as the standard input and
standard output) in poke is not trivial.  We considered two
possibilities:

Option 1: to handle stdin and stdout in a special way at the Poke
language level.  This basically would mean to alter the semantics of the
mapping operator: mapping in a stream IO space "advances" the file
position.

A little Poke program that would process TCP packets from stdin and
writing them out in stdout, using Option 1, would look like:

   #!/usr/bin/poke -L
   !#
   
   load tcp;
   
   try do_stuff (Tcp_Packet @ stdin : 0#B);
   until E_eof;

We quickly discovered that Option 1 is not really feasible: "advancing"
the file position implicitly basically means adjusting offsets, and that
totally breaks the mapping.  Also, it is not a very "pokish" way of
resolving the problem.  So we had to think about an alternative.

Option 2: to handle stdin and stdout (and other stream-like devices) as
regular IO spaces.  This basically implies the need to "remember" the
data already read from read-only streams (using either a memory buffer
or a temporary file in the filesystem) and to provide a way for the user
to "forget" that data when it is no longer needed.

This new operation has the following form:

   forget IOS, OFFSET; -> From 0#B..OFFSET in IOS, "forget" it.

If a reference to a value mapped in that range is done, then the
peek/poke PVM instructions should raise an E_eof exception (meaning in
this case Early Of File, instead of End Of File :D).  Forgetting an IO
space that is backed by a regular file (and not a stream-like device) is
a nop.  Likewise, forgetting a write-only stream IO space is also a nop.

Making streams to look like regular IO spaces also requires a way to
append to them, in the case of write-only streams.  For that purpose we
decided to add a new built-in function `iosize' that, given the ID of an
IO space, returns the offset past the end of the space.  This is the
offset to be used in a lhs mapping.

But what should `iosize' return if it is invoked in a read-only stream?
Should it raise an exception?  Bruno thinks we should raise an exception
for stdin because otherwise it may reveal some details on the buffering
strategy.  John and Darshit agrees.

The TCP filter above, using Option 2, would look like:

   defvar offset = 0#B;
   
   try
   {
     forget stdin, offset;
   
     defvar b = byte @ stdin : offset;
   
     if (b < 80)
        byte @ stdout : iosize (stdout) = b;

     offset = offset + packet'size;
   }
   until E_eof;

Since appending to an IO space will be a common operation when handling
stream-like spaces, we considered to add some syntax sugar, something
like:

  byte @ stdout :+ = b;


                              [Unicode]

How to best support Unicode in poke?  We concluded that mimicking the C
support (with its support of "wide" chars and strings) is not a good
idea.  We will be splitting the support in several pickles:
      
  unicode.pk (also handling ucs encodings)
  utf8.pk
  utf16.pk

Additionally to provide suitable Poke types (like for an UTF-8
character) we will want to implement additional functionality (we used
the GNU libunistring API as a base):

- display width (for printing)

- We don't need explicit check functions because that logic shall be
  implemented in the UTF8 type definitions (constraints.)
  
- Conversion functions (utf* -> utf*)

- mblen functions are not needed because the logic is implemented as
  part of the mapping.

- ditto for the *cpy functions.
- ditto for the *move functions.
- ditto for the *mbsnlen functions.
- ditto for *next and *prev.
- ditto for *strlen.
- Comparison functions are useful (for sorting for example.)
- strstr for unicode strings.

                           [.source and source]

We also discussed about the need to have a way to "load" (or source)
pickles at the Poke language level.  This way, we will be able to write
Poke programs like:

   #!/usr/bin/poke -L
   !#
   
   load elf;
   
   ... operate on ELF stuff ...

This would have the same semantics than the currently available `.load'
dot-command.  At this point Bruno pointed out that if, once a given
pickle has been loaded, loading it again is a nop, then calling the
operation "load" is confusing.  Darshit suggested using the name
"source" instead.  After some discussion we agreed to use "source" for
both the dot-command and the language-level construction, with the same
semantics than the current dot-command.

At some point (surely after the release) we will be adding a modules
system to Poke.  But for the time being "sourcing" pickles shall be
enough.

                        [we got a bug tracker!]

We finally decided on getting ourselves a bug tracker, and concluded
bugzilla would be a good option: it allows programmatic access, email
notifications, and provides a good web interface for people enjoying
such things.  So we went ahead and requested the addition of poke as a
product in the sourceware bugzilla (which also manages the bugs of other
GNU programs such as glibc and binutils).  The sourceware overseers
created the product almost immediately (thanks Frank, you are the best!)
so we can already use it at our pleasure.

                   [looking forward to meet again]

All in all, this was a great experience.  I'm looking forward to meet
again with my fellow pokers, which will happen at FOSDEM in a few weeks!
I will definitely organize another Poke-Conf colocated with the GNU
Hackers Meeting in Hamburg this summer.  At that point we will have
already released poke 1.0 so we will have a great excuse to make a
party... or so I hope! :)

Happy poking!