Re: A curious bug...

[Dr . Jürgen Sauermann]

    Hi Emmanuel,
    
    I cant quite explain this fully, but if I copy-and-paste your string
    below
    and print it in hex, then:
    
    eedjsa@server68:~/apl-1.8/src$ echo 'Z ← L
        foo R' | od -t x1z
      0000000 5a 20 e2 86 90 20 4c 20 66 6f 6f 20 52 0a       
        >Z ... L foo R.<
      0000016
      
    which suggests that the sequence e2 86 90
    (which, by chance, is
    the UTF8 encoding of U+2190 aka. ←) made it into the tokenizer
    of GNU APL (which then correctly complains).
    
    In other words, it very much looks to me as if some conversion from
    UTF8 encoding to Unicode (aka. UCS) is missing on the way of your
    header line from emacs to APL.
    
    Unfortunately I know very little as to how emacs works,
    maybe
    Elias knows better how to troubleshoot this.
    
    Best Regards,
    Jürgen

On 5/29/23 19:42, Emmanuel Charpentier wrote:

Le lundi 29 mai 2023 à 18:07 +0200, Dr. Jürgen Sauermann a écrit :

Hi,

the make distclean may overwrite some Makefile.in files
which are shipped with GNU APL.

Normally your ./configure below (without distclean)
in a fresh (yours is now broken) svn co should do it.
what you want.

Done. Not fun...

Now trying in emacs's gnu-apl :

This is the gnu-apl-mode interactive buffer.

To toggle keyboard help, call M-x gnu-apl-show-keyboard (C-c C-k by default).
APL symbols are bound to the standard keys with the Super key. You can also
activate the APL-Z input method (M-x toggle-input-method or C-\) which
allows you to input APL symbols by prefixing the key with a "." (period).

There are several customisation options that can be set.
Click the link or run M-x customize-group RET gnu-apl to set up.

To disable this message, set gnu-apl-show-tips-on-start to nil.

Network listener started. Connection information: mode:tcp addr:33853

                    ______ _   __ __  __    ___     ____   __ 
                   / ____// | / // / / /   /   |   / __ \ / / 
                  / / __ /  |/ // / / /   / /| |  / /_/ // /  
                 / /_/ // /|  // /_/ /   / ___ | / ____// /___
                 \____//_/ |_/ \____/   /_/  |_|/_/    /_____/

                  Welcome to GNU APL version 1.8 / SVN: 1699M

                 Copyright © 2008-2023  Dr. Jürgen Sauermann
                       Banner by FIGlet: www.figlet.org

                This program comes with ABSOLUTELY NO WARRANTY;
                  for details run: /usr/local/bin/apl --gpl.

     This program is free software, and you are welcome to redistribute it
         according to the GNU Public License (GPL) version 3 or later.

      Illegal : in immediate execution+
      ]log 6
    Logging facility 6:  ...    tokenization                 is now ON 
      ∇ Z ← L foo R

gnu-apl opens a window called *gnu-apl edit function* with my header. I fill it ; resulting buffer :

∇Z ← L foo R
  Z ← L + R
∇

Typing this seems normal : I typed the second line in column 1, emacs indented it in col 3 and moves the cursor to the tird line in line 3, column 3, where I type ∇ ; the following "Return" makes emacs move the closing ∇ to column 1 and the cursor to line 4 column 1.

That's where I type C-c C-c ; which prompts emacs to type

      tokenize: input[13] is: «Z ← L foo R»
  tokenize(13 chars) sees [tag TOK_SYMBOL «Z»] Z ← L foo R
tokenize_symbol() : 13
  tokenize(12 chars) sees [tag TOK_SPACE « »]  ← L foo R
  tokenize(11 chars) sees [tag TOK_VOID «\342»] ← L foo R
Unknown APL character: \342 (U+FFFFFFE2)

in the *gnu-apl buffer.

Correcting the *gnu-apl edit function* buffer to :

∇Z←L foo R
Z←L+R
∇

and typing C-c C-c gets me :

tokenize: input[11] is: «Z←L foo R»
  tokenize(11 chars) sees [tag TOK_SYMBOL «Z»] Z←L foo R
tokenize_symbol() : 11
  tokenize(10 chars) sees [tag TOK_VOID «\342»] ←L foo R
Unknown APL character: \342 (U+FFFFFFE2)

This is where I throw the towel and kills the *gnu-apl buffer. emacs prompts me to kill the interpreter, which I approve.

Restarting, I open gnu-apl in one frame and a temporary *AplTmp* buffer in another frame, where I type :

⍝ test buffer
∇ Z ← L foo R
  Z ← L + R
∇

During this edition, emacs reacts exactly as it did in the del-editor buffer of the first trial.

Typing C-c C-l (= gnu-apl-interactive-send-buffer) gets me :

      tokenize: input[2] is: «⊣⍬»
  tokenize(2 chars) sees [tag TOK_F2_LEFT «⊣»] ⊣⍬
tokenize_function(⊣⍬)
  tokenize(1 chars) sees [tag TOK_F0_ZILDE «⍬»] ⍬
tokenize_function(⍬)
tokenize() done (no error).
   └── tos[2] is: ⊣ VALUE1«≡⊏0⊐»
tokenize: input[11] is: «Z ← L foo R»
  tokenize(11 chars) sees [tag TOK_SYMBOL «Z»] Z ← L foo R
tokenize_symbol() : 11
  tokenize(10 chars) sees [tag TOK_SPACE « »]  ← L foo R
  tokenize(9 chars) sees [tag TOK_ASSIGN «←»] ← L foo R
  tokenize(8 chars) sees [tag TOK_SPACE « »]  L foo R
  tokenize(7 chars) sees [tag TOK_SYMBOL «L»] L foo R
tokenize_symbol() : 7
  tokenize(6 chars) sees [tag TOK_SPACE « »]  foo R
  tokenize(5 chars) sees [tag TOK_SYMBOL «f»] foo R
tokenize_symbol() : 5
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L 'foo 'R
tokenize: input[0] is: «»
tokenize() done (no error).
   └── tos[0] is:
tokenize: input[11] is: «  Z ← L + R»
  tokenize(11 chars) sees [tag TOK_SPACE « »]   Z ← L + R
  tokenize(10 chars) sees [tag TOK_SPACE « »]  Z ← L + R
  tokenize(9 chars) sees [tag TOK_SYMBOL «Z»] Z ← L + R
tokenize_symbol() : 9
  tokenize(8 chars) sees [tag TOK_SPACE « »]  ← L + R
  tokenize(7 chars) sees [tag TOK_ASSIGN «←»] ← L + R
  tokenize(6 chars) sees [tag TOK_SPACE « »]  L + R
  tokenize(5 chars) sees [tag TOK_SYMBOL «L»] L + R
tokenize_symbol() : 5
  tokenize(4 chars) sees [tag TOK_SPACE « »]  + R
  tokenize(3 chars) sees [tag TOK_F12_PLUS «+»] + R
tokenize_function(+ R)
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L + 'R
tokenize: input[11] is: «Z ← L foo R»
  tokenize(11 chars) sees [tag TOK_SYMBOL «Z»] Z ← L foo R
tokenize_symbol() : 11
  tokenize(10 chars) sees [tag TOK_SPACE « »]  ← L foo R
  tokenize(9 chars) sees [tag TOK_ASSIGN «←»] ← L foo R
  tokenize(8 chars) sees [tag TOK_SPACE « »]  L foo R
  tokenize(7 chars) sees [tag TOK_SYMBOL «L»] L foo R
tokenize_symbol() : 7
  tokenize(6 chars) sees [tag TOK_SPACE « »]  foo R
  tokenize(5 chars) sees [tag TOK_SYMBOL «f»] foo R
tokenize_symbol() : 5
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L 'foo 'R
tokenize: input[0] is: «»
tokenize() done (no error).
   └── tos[0] is:
tokenize: input[11] is: «  Z ← L + R»
  tokenize(11 chars) sees [tag TOK_SPACE « »]   Z ← L + R
  tokenize(10 chars) sees [tag TOK_SPACE « »]  Z ← L + R
  tokenize(9 chars) sees [tag TOK_SYMBOL «Z»] Z ← L + R
tokenize_symbol() : 9
  tokenize(8 chars) sees [tag TOK_SPACE « »]  ← L + R
  tokenize(7 chars) sees [tag TOK_ASSIGN «←»] ← L + R
  tokenize(6 chars) sees [tag TOK_SPACE « »]  L + R
  tokenize(5 chars) sees [tag TOK_SYMBOL «L»] L + R
tokenize_symbol() : 5
  tokenize(4 chars) sees [tag TOK_SPACE « »]  + R
  tokenize(3 chars) sees [tag TOK_F12_PLUS «+»] + R
tokenize_function(+ R)
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L + 'R
tokenize: input[11] is: «  Z ← L + R»
  tokenize(11 chars) sees [tag TOK_SPACE « »]   Z ← L + R
  tokenize(10 chars) sees [tag TOK_SPACE « »]  Z ← L + R
  tokenize(9 chars) sees [tag TOK_SYMBOL «Z»] Z ← L + R
tokenize_symbol() : 9
  tokenize(8 chars) sees [tag TOK_SPACE « »]  ← L + R
  tokenize(7 chars) sees [tag TOK_ASSIGN «←»] ← L + R
  tokenize(6 chars) sees [tag TOK_SPACE « »]  L + R
  tokenize(5 chars) sees [tag TOK_SYMBOL «L»] L + R
tokenize_symbol() : 5
  tokenize(4 chars) sees [tag TOK_SPACE « »]  + R
  tokenize(3 chars) sees [tag TOK_F12_PLUS «+»] + R
tokenize_function(+ R)
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L + 'R
tokenize: input[11] is: «  Z ← L + R»
  tokenize(11 chars) sees [tag TOK_SPACE « »]   Z ← L + R
  tokenize(10 chars) sees [tag TOK_SPACE « »]  Z ← L + R
  tokenize(9 chars) sees [tag TOK_SYMBOL «Z»] Z ← L + R
tokenize_symbol() : 9
  tokenize(8 chars) sees [tag TOK_SPACE « »]  ← L + R
  tokenize(7 chars) sees [tag TOK_ASSIGN «←»] ← L + R
  tokenize(6 chars) sees [tag TOK_SPACE « »]  L + R
  tokenize(5 chars) sees [tag TOK_SYMBOL «L»] L + R
tokenize_symbol() : 5
  tokenize(4 chars) sees [tag TOK_SPACE « »]  + R
  tokenize(3 chars) sees [tag TOK_F12_PLUS «+»] + R
tokenize_function(+ R)
  tokenize(2 chars) sees [tag TOK_SPACE « »]  R
  tokenize(1 chars) sees [tag TOK_SYMBOL «R»] R
tokenize_symbol() : 1
tokenize() done (no error).
   └── tos[5] is: 'Z ← 'L + 'R
         └── tos[5] is: 'Z ← 'L + 'R

The cursor is still immediately after the end of the last line. Typing a "Return" to get the prompt dets me :

tokenize: input[27] is: «└── tos[5] is: 'Z ← 'L + 'R»
  tokenize(27 chars) sees [tag TOK_END «└»] └── tos[5] is: 'Z ← 'L + ...
SYNTAX ERROR+

Testing the function :

      2 foo 3
tokenize: input[7] is: «2 foo 3»
  tokenize(7 chars) sees [tag TOK_NUMERIC «2»] 2 foo 3
tokenize_number(2 foo 3)
int_digits.size():   '1' at Tokenizer.cc:951
int_digits:          '2' at Tokenizer.cc:952
fract_digits:        '' at Tokenizer.cc:953
expo_digits:         '' at Tokenizer.cc:954
  tokenize_number: integer 2
  tokenize(6 chars) sees [tag TOK_SPACE « »]  foo 3
  tokenize(5 chars) sees [tag TOK_SYMBOL «f»] foo 3
tokenize_symbol() : 5
  tokenize(2 chars) sees [tag TOK_SPACE « »]  3
  tokenize(1 chars) sees [tag TOK_NUMERIC «3»] 3
tokenize_number(3)
int_digits.size():   '1' at Tokenizer.cc:951
int_digits:          '3' at Tokenizer.cc:952
fract_digits:        '' at Tokenizer.cc:953
expo_digits:         '' at Tokenizer.cc:954
  tokenize_number: integer 3
tokenize() done (no error).
   └── tos[3] is: INTEGER 'foo INTEGER
5

shows that the function has been correctly interpreted and created.

Best Regards,

As usual, logs attached.

HTH,

--
Emmanuel Charpentier

[ Snip... ]