Re: SMIE

[Matt DeBoard]

In general I’m having a hard time connecting the dots between the BNF
grammar table creation, the smie-rules (i.e. :before, :after, etc.),
tokenization, indentation, and so forth, and how it all comes together
to make this indentation machine work.

It’s not that the manual is poorly written. In fact between the manual
and the code comments it’s pretty comprehensive. It’s just a high
volume of new information. I think it’ll take me awhile to ingest
everything, but when I do I’d be glad to contribute back.

Regarding the specific issue I mentioned, I closed that out tonight.
There was some overly “greedy” definitions in the syntax table, so
there’s that. I’m slowly starting to pare things away. The bit you
wrote about :list-intro is interesting. When you say that it sees two
or more concatenated expressions, how does that tie in to the BNF
grammar definitions?

On Wed, Jul 9, 2014 at 11:02 PM, Stefan Monnier
<address@hidden> wrote:
>> Hi there. As the subject line says I’m writing for help with SMIE.
>
> Cool!
>
>> I am currently working on elixir-mode
>> <https://github.com/elixir-lang/emacs-elixir>, having (apparently) taking
>> over the mode as the latest in a line of contributors.
>
> I'd love to include this in GNU ELPA.  Interested?
>
>> Specifically I’m having trouble understanding the mental model for how
>> tokenisation & indentation works.  For example, in this
>> <https://github.com/elixir-lang/emacs-elixir/issues/18> issue, indentation
>> errors seem to crop up only after separating lines of code with blank
>> lines.
>> I have spent, seriously, hundreds of hours trying to sort out what’s
>> happening here and I am at my wits’ end.
>
> IIUC, Elixir syntax does not treat all whitespace as "irrelevant",
> contrary to the default tokenizer of SMIE.
>
>> Does this issue ring any bells with issues you’ve dealt with in
>> the past?
>
> Yes, indeed.  Octave and sh are two other languages that use SMIE and
> where some whitespace is syntactically significant.
>
> What you need to do is to change the tokenizer so that instead of
> skipping all whitespace, it turns the syntactically-significant
> whitespace into a token (you can name it any way you like; in the above
> languages, it turns out to be syntactically equivalent to a semi-colon,
> so we call it ";").
>
> I know absolutely nothing about Elixir or its syntax, so I can't give
> you specific details, but you can look at octave.el and sh-script.el
> for examples.  Feel free to email me back with more details if you need
> further help.
>
>> Final question, how is it determined if a token is a :list-intro token?
>
> Not sure I understand the question.  The issue is for the indentation
> rules, when it sees two (or more) concatenated expressions (e.g. "exp1
> exp2"), should it assume that exp2 is something like an argument to the
> exp1 function (and hence exp2 (and exp3, ...) should be indented like
> a function argument) or are all those "expressions" just a list, where
> the first is not more special than the second?
> This usually depends on the context.  E.g. in a situation like
>
>    fun x1 x2 x3 =>
>
> x2 is not an argument passed to the function x1;  Instead x1, x2, and x3
> are "siblings" and should be indented to the same level.  So to decide
> how to indent x2 and x3 w.r.t x1, SMIE calls the smie-rule-function with
> (:list-intro . "fun") so smie-rule-function can tell it that "fun"
> introduces a *list* of "things" rather than being followed by a "normal
> expression".
>
> Does that make more sense?
>
>> I have read the SMIE manual ten times, at least, but I’m really
>> struggling.  I would truly appreciate your help.
>
> I'm not very good at writing manuals, sorry.  But I promise to do my
> best to help you get SMIE working well.  In return, I would appreciate
> if you could help me improve the doc by giving, if not actual patches,
> at least suggestions of how to rewrite the doc, or what to add to it
> (usually, you can only make such suggestions after you finally
> understand what's going on, and at the same time it's
> important/necessary/useful to try and remember what it was that you
> didn't understand).
>
>
>         Stefan