Re: SMIE examples or guides

[Björn Lindqvist]

I managed to write the forward and backward tokenization functions:

(defun factor-smie-token (dir)
  (pcase dir
    ('forward (forward-comment (point-max)))
    ('backward (forward-comment (- (point)))))
  (let ((tok (buffer-substring-no-properties
              (point)
              (let ((syntax "w_\\\""))
                (pcase dir
                  ('forward (skip-syntax-forward syntax))
                  ('backward (skip-syntax-backward syntax)))
                (point)))))
    ;; Normalizes different indent starters.
    (cond ((string-match factor-smie-indents-regex tok) ":")
          (t tok))))

(defun factor-smie-forward-token ()
  (factor-smie-token 'forward))

(defun factor-smie-backward-token ()
  (factor-smie-token 'backward))

It works in 99% of the cases. But since almost any character can be
part of a token, it doesn't work perfectly. E.g te]st(ab3 would be a
perfectly valid variable name. I tried changing the syntax to
"w_()\\\"" and that fixes the tokenization but then I lose the useful
automatic indentation smie adds to opening- and closing bracket
characters. E.g I'm happy that smie indents:

[
    neat
    {
        nice
    }
    (
        good
    )
]

It's very nice. But the following two lines are not right:

hi[there
   two

They should be:

hi[there
two


2016-07-01 9:13 GMT+02:00 Stefan Monnier <monnier@iro.umontreal.ca>:
>> Many thanks! That appears to work. But there is one problem. In a
>> buffer with just this content:
>
>>     [
>>     \
>>     ]
>
>> I get the error: (error "Bumped into unknown token")
>
> You should either give a different syntax to the \ char in the
> syntax-table, or change the :forward-token and :backward-token functions
> so they do something with \
>
>>       (smie-bnf->prec2
>>        '((exp (("HELLO" "HALLO" "CIAO" "SALUT") exp "BYE"))))))
>
> One way to do that is:
>
>        (smie-bnf->prec2
>         '((exp ("HELLO" exp "BYE")
>                ("HALLO" exp "BYE")
>                ("CIAO" exp "BYE")
>                ("SALUT" exp "BYE"))))))
>
> Another is to change the :forward-token and :backward-token functions so
> they return the same string (e.g. "HELLO") when tokenizing any one of those.
>
>
>         Stefan



-- 
mvh/best regards Björn Lindqvist