bug#8218: 23.2; scan-sexps sees inconsistent syntax

[Dima Kogan]

<#part type="text/x-perl" filename="/tmp/tst3.pm" disposition=inline 
description="bug demo">
<#/part>

I am seeing (scan-sexps) show inconsistent behavior depending on WHEN it
is evaluated. If the attached sample file is opened, the point placed at
the first '{', and (scan-sexps (point) 1) evaluated, the returned
position is at times of the first '}' (wrong) and at other times of the
second '}' (right). If the evaluation is done right when the file is
opened, I observe the wrong behavior. This is reliably triggered with

$ emacs -Q /tmp/tst3.pm --eval '(goto-char (scan-sexps 5 1))'

Any subsequent evaluation of the same form in that buffer produces the
correct behavior. If I turn off font-lock-mode, I consistently get
incorrect behavior. This has wide ramifications with occasional
incorrect indentation and hide-show operation, for instance.

I looked through the source a bit, and I see an inconsistency in the
syntax code returned by the
   code = SYNTAX_WITH_MULTIBYTE_CHECK (c)

line in scan_lists() in syntax.c . When things work, the '$' is
interpreted as punctuation and the following '{' as a block-open. When
things don't work, the '$' is a quote and the '{' is skipped
altogether:

| character | incorrect syntax | correct syntax |
|-----------+------------------+----------------|
| =         | Spunct           | Spunct         |
|           | Swhitespace      | Swhitespace    |
| $         | Scharquote       | Spunct         |
| {         |                  | Sopen          |
| P         | Sword            | Sword          |

syntax.c is a bit difficult to dive into, but it would be great if
someone who has experience with it would take a look.



In GNU Emacs 23.2.1 (x86_64-pc-linux-gnu, GTK+ Version 2.20.1)
 of 2010-12-11 on brahms, modified by Debian
Windowing system distributor `The X.Org Foundation', version 11.0.10707000
configured using `configure  '--build' 'x86_64-linux-gnu' '--build' 
'x86_64-linux-gnu' '--prefix=/usr' '--sharedstatedir=/var/lib' 
'--libexecdir=/usr/lib' '--localstatedir=/var/lib' '--infodir=/usr/share/info' 
'--mandir=/usr/share/man' '--with-pop=yes' 
'--enable-locallisppath=/etc/emacs23:/etc/emacs:/usr/local/share/emacs/23.2/site-lisp:/usr/local/share/emacs/site-lisp:/usr/share/emacs/23.2/site-lisp:/usr/share/emacs/site-lisp:/usr/share/emacs/23.2/leim'
 '--with-x=yes' '--with-x-toolkit=gtk' '--with-toolkit-scroll-bars' 
'build_alias=x86_64-linux-gnu' 'CFLAGS=-DDEBIAN -g -O2' 'LDFLAGS=-g' 
'CPPFLAGS=''

Important settings:
  value of $LC_ALL: nil
  value of $LC_COLLATE: nil
  value of $LC_CTYPE: nil
  value of $LC_MESSAGES: nil
  value of $LC_MONETARY: nil
  value of $LC_NUMERIC: nil
  value of $LC_TIME: nil
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: nil
  locale-coding-system: utf-8-unix
  default enable-multibyte-characters: t

Major mode: Perl

Minor modes in effect:
  tooltip-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent input:
<up> <up> M-: <help-echo> <down-mouse-2> <mouse-2> 
<return> <up> <up> <up> <up> <up> <up> <up> <down> 
M-x r e p o r t SPC b u g <return> s c a <backspace> 
a n - s e x p s SPC g e n e r a t e s SPC i n c o n 
s i s t e n t SPC s y n t a x C-SPC <home> M-w C-g 
C-g C-g <up> <up> <right> <right> <right> <right> <right> 
<right> <right> <right> <right> <right> <right> <right> 
<right> M-: ( p o i n t ) <return> <up> M-x <up> <return> 
C-g M-x <up> <return>

Recent messages:
Loading /etc/emacs/site-start.d/50psvn.el (source)...done
Loading /etc/emacs23/site-start.d/50vm-init.el (source)...done
Loading /etc/emacs/site-start.d/50w3m-el.el (source)...done
Loading /etc/emacs/site-start.d/51semi.el (source)...done
Loading /etc/emacs/site-start.d/51speedbar.el (source)...done
Loading /etc/emacs/site-start.d/65wl-beta.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
212 (#o324, #xd4)
Quit [3 times]
92 (#o134, #x5c)
Quit

Load-path shadows:
/usr/share/emacs/23.2/site-lisp/auctex/tex-bar hides 
/usr/share/emacs/site-lisp/auctex/tex-bar
/usr/share/emacs/23.2/site-lisp/auctex/context-nl hides 
/usr/share/emacs/site-lisp/auctex/context-nl
/usr/share/emacs/23.2/site-lisp/auctex/tex-buf hides 
/usr/share/emacs/site-lisp/auctex/tex-buf
/usr/share/emacs/23.2/site-lisp/auctex/texmathp hides 
/usr/share/emacs/site-lisp/auctex/texmathp
/usr/share/emacs/23.2/site-lisp/auctex/tex hides 
/usr/share/emacs/site-lisp/auctex/tex
/usr/share/emacs/23.2/site-lisp/auctex/tex-mik hides 
/usr/share/emacs/site-lisp/auctex/tex-mik
/usr/share/emacs/23.2/site-lisp/auctex/tex-fptex hides 
/usr/share/emacs/site-lisp/auctex/tex-fptex
/usr/share/emacs/23.2/site-lisp/auctex/bib-cite hides 
/usr/share/emacs/site-lisp/auctex/bib-cite
/usr/share/emacs/23.2/site-lisp/auctex/tex-info hides 
/usr/share/emacs/site-lisp/auctex/tex-info
/usr/share/emacs/23.2/site-lisp/auctex/tex-jp hides 
/usr/share/emacs/site-lisp/auctex/tex-jp
/usr/share/emacs/23.2/site-lisp/auctex/tex-font hides 
/usr/share/emacs/site-lisp/auctex/tex-font
/usr/share/emacs/23.2/site-lisp/auctex/toolbar-x hides 
/usr/share/emacs/site-lisp/auctex/toolbar-x
/usr/share/emacs/23.2/site-lisp/auctex/latex hides 
/usr/share/emacs/site-lisp/auctex/latex
/usr/share/emacs/23.2/site-lisp/auctex/font-latex hides 
/usr/share/emacs/site-lisp/auctex/font-latex
/usr/share/emacs/23.2/site-lisp/auctex/tex-fold hides 
/usr/share/emacs/site-lisp/auctex/tex-fold
/usr/share/emacs/23.2/site-lisp/auctex/context hides 
/usr/share/emacs/site-lisp/auctex/context
/usr/share/emacs/23.2/site-lisp/auctex/context-en hides 
/usr/share/emacs/site-lisp/auctex/context-en
/usr/share/emacs/23.2/site-lisp/auctex/multi-prompt hides 
/usr/share/emacs/site-lisp/auctex/multi-prompt
/usr/share/emacs/23.2/site-lisp/auctex/tex-style hides 
/usr/share/emacs/site-lisp/auctex/tex-style
/usr/local/share/emacs/site-lisp/imaxima-autoconf-variables hides 
/usr/share/emacs/23.2/site-lisp/maxima/imaxima-autoconf-variables
/usr/local/share/emacs/site-lisp/imaxima hides 
/usr/share/emacs/23.2/site-lisp/maxima/imaxima
/usr/local/share/emacs/site-lisp/imath hides 
/usr/share/emacs/23.2/site-lisp/maxima/imath
/usr/share/emacs/site-lisp/autoconf/autotest-mode hides 
/usr/share/emacs/site-lisp/autotest-mode
/usr/share/emacs/23.2/site-lisp/magit hides /usr/share/emacs/site-lisp/magit
/usr/share/emacs/23.2/site-lisp/cscope/xcscope hides 
/usr/share/emacs/site-lisp/xcscope
/usr/share/emacs/23.2/site-lisp/cmake-data/cmake-mode hides 
/usr/share/emacs/site-lisp/cmake-mode
/usr/share/emacs/23.2/site-lisp/debian-startup hides 
/usr/share/emacs/site-lisp/debian-startup
/usr/share/emacs23/site-lisp/semi/pgg-pgp5 hides 
/usr/share/emacs/23.2/lisp/pgg-pgp5
/usr/share/emacs23/site-lisp/speedbar/sb-image hides 
/usr/share/emacs/23.2/lisp/sb-image
/usr/share/emacs23/site-lisp/semi/pgg hides /usr/share/emacs/23.2/lisp/pgg
/usr/share/emacs23/site-lisp/semi/pgg-parse hides 
/usr/share/emacs/23.2/lisp/pgg-parse
/usr/share/emacs23/site-lisp/cedet-common/ezimage hides 
/usr/share/emacs/23.2/lisp/ezimage
/usr/share/emacs23/site-lisp/flim/md4 hides /usr/share/emacs/23.2/lisp/md4
/usr/share/emacs23/site-lisp/semi/pgg-gpg hides 
/usr/share/emacs/23.2/lisp/pgg-gpg
/usr/share/emacs23/site-lisp/speedbar/speedbar hides 
/usr/share/emacs/23.2/lisp/speedbar
/usr/share/emacs23/site-lisp/flim/sha1 hides /usr/share/emacs/23.2/lisp/sha1
/usr/share/emacs23/site-lisp/semi/pgg-pgp hides 
/usr/share/emacs/23.2/lisp/pgg-pgp
/usr/share/emacs23/site-lisp/flim/hex-util hides 
/usr/share/emacs/23.2/lisp/hex-util
/usr/share/emacs23/site-lisp/speedbar/dframe hides 
/usr/share/emacs/23.2/lisp/dframe
/usr/share/emacs23/site-lisp/semi/pgg-def hides 
/usr/share/emacs/23.2/lisp/pgg-def
/usr/share/emacs23/site-lisp/dictionaries-common/flyspell hides 
/usr/share/emacs/23.2/lisp/textmodes/flyspell
/usr/share/emacs23/site-lisp/dictionaries-common/ispell hides 
/usr/share/emacs/23.2/lisp/textmodes/ispell
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-inf hides 
/usr/share/emacs/23.2/lisp/progmodes/octave-inf
/usr/share/emacs/23.2/site-lisp/octave3.1-emacsen/octave-mod hides 
/usr/share/emacs/23.2/lisp/progmodes/octave-mod
/usr/share/emacs23/site-lisp/org-mode/org-archive hides 
/usr/share/emacs/23.2/lisp/org/org-archive
/usr/share/emacs23/site-lisp/org-mode/org-mobile hides 
/usr/share/emacs/23.2/lisp/org/org-mobile
/usr/share/emacs23/site-lisp/org-mode/org-ascii hides 
/usr/share/emacs/23.2/lisp/org/org-ascii
/usr/share/emacs23/site-lisp/org-mode/org-clock hides 
/usr/share/emacs/23.2/lisp/org/org-clock
/usr/share/emacs23/site-lisp/org-mode/org-gnus hides 
/usr/share/emacs/23.2/lisp/org/org-gnus
/usr/share/emacs23/site-lisp/org-mode/org-indent hides 
/usr/share/emacs/23.2/lisp/org/org-indent
/usr/share/emacs23/site-lisp/org-mode/org-wl hides 
/usr/share/emacs/23.2/lisp/org/org-wl
/usr/share/emacs23/site-lisp/org-mode/org-exp-blocks hides 
/usr/share/emacs/23.2/lisp/org/org-exp-blocks
/usr/share/emacs23/site-lisp/org-mode/org-list hides 
/usr/share/emacs/23.2/lisp/org/org-list
/usr/share/emacs23/site-lisp/org-mode/org-w3m hides 
/usr/share/emacs/23.2/lisp/org/org-w3m
/usr/share/emacs23/site-lisp/org-mode/org-colview hides 
/usr/share/emacs/23.2/lisp/org/org-colview
/usr/share/emacs23/site-lisp/org-mode/org-freemind hides 
/usr/share/emacs/23.2/lisp/org/org-freemind
/usr/share/emacs23/site-lisp/org-mode/org-macs hides 
/usr/share/emacs/23.2/lisp/org/org-macs
/usr/share/emacs23/site-lisp/org-mode/org-crypt hides 
/usr/share/emacs/23.2/lisp/org/org-crypt
/usr/share/emacs23/site-lisp/org-mode/org-mhe hides 
/usr/share/emacs/23.2/lisp/org/org-mhe
/usr/share/emacs23/site-lisp/org-mode/org-table hides 
/usr/share/emacs/23.2/lisp/org/org-table
/usr/share/emacs23/site-lisp/org-mode/org hides 
/usr/share/emacs/23.2/lisp/org/org
/usr/share/emacs23/site-lisp/org-mode/org-bibtex hides 
/usr/share/emacs/23.2/lisp/org/org-bibtex
/usr/share/emacs23/site-lisp/org-mode/org-src hides 
/usr/share/emacs/23.2/lisp/org/org-src
/usr/share/emacs23/site-lisp/org-mode/org-compat hides 
/usr/share/emacs/23.2/lisp/org/org-compat
/usr/share/emacs23/site-lisp/org-mode/org-publish hides 
/usr/share/emacs/23.2/lisp/org/org-publish
/usr/share/emacs23/site-lisp/org-mode/org-id hides 
/usr/share/emacs/23.2/lisp/org/org-id
/usr/share/emacs23/site-lisp/org-mode/org-feed hides 
/usr/share/emacs/23.2/lisp/org/org-feed
/usr/share/emacs23/site-lisp/org-mode/org-habit hides 
/usr/share/emacs/23.2/lisp/org/org-habit
/usr/share/emacs23/site-lisp/org-mode/org-rmail hides 
/usr/share/emacs/23.2/lisp/org/org-rmail
/usr/share/emacs23/site-lisp/org-mode/org-plot hides 
/usr/share/emacs/23.2/lisp/org/org-plot
/usr/share/emacs23/site-lisp/org-mode/org-protocol hides 
/usr/share/emacs/23.2/lisp/org/org-protocol
/usr/share/emacs23/site-lisp/org-mode/org-faces hides 
/usr/share/emacs/23.2/lisp/org/org-faces
/usr/share/emacs23/site-lisp/org-mode/org-latex hides 
/usr/share/emacs/23.2/lisp/org/org-latex
/usr/share/emacs23/site-lisp/org-mode/org-jsinfo hides 
/usr/share/emacs/23.2/lisp/org/org-jsinfo
/usr/share/emacs23/site-lisp/org-mode/org-xoxo hides 
/usr/share/emacs/23.2/lisp/org/org-xoxo
/usr/share/emacs23/site-lisp/org-mode/org-vm hides 
/usr/share/emacs/23.2/lisp/org/org-vm
/usr/share/emacs23/site-lisp/org-mode/org-mouse hides 
/usr/share/emacs/23.2/lisp/org/org-mouse
/usr/share/emacs23/site-lisp/org-mode/org-timer hides 
/usr/share/emacs/23.2/lisp/org/org-timer
/usr/share/emacs23/site-lisp/org-mode/org-footnote hides 
/usr/share/emacs/23.2/lisp/org/org-footnote
/usr/share/emacs23/site-lisp/org-mode/org-html hides 
/usr/share/emacs/23.2/lisp/org/org-html
/usr/share/emacs23/site-lisp/org-mode/org-inlinetask hides 
/usr/share/emacs/23.2/lisp/org/org-inlinetask
/usr/share/emacs23/site-lisp/org-mode/org-attach hides 
/usr/share/emacs/23.2/lisp/org/org-attach
/usr/share/emacs23/site-lisp/org-mode/org-agenda hides 
/usr/share/emacs/23.2/lisp/org/org-agenda
/usr/share/emacs23/site-lisp/org-mode/org-datetree hides 
/usr/share/emacs/23.2/lisp/org/org-datetree
/usr/share/emacs23/site-lisp/org-mode/org-bbdb hides 
/usr/share/emacs/23.2/lisp/org/org-bbdb
/usr/share/emacs23/site-lisp/org-mode/org-irc hides 
/usr/share/emacs/23.2/lisp/org/org-irc
/usr/share/emacs23/site-lisp/org-mode/org-mew hides 
/usr/share/emacs/23.2/lisp/org/org-mew
/usr/share/emacs23/site-lisp/org-mode/org-mac-message hides 
/usr/share/emacs/23.2/lisp/org/org-mac-message
/usr/share/emacs23/site-lisp/org-mode/org-docbook hides 
/usr/share/emacs/23.2/lisp/org/org-docbook
/usr/share/emacs23/site-lisp/org-mode/org-info hides 
/usr/share/emacs/23.2/lisp/org/org-info
/usr/share/emacs23/site-lisp/org-mode/org-exp hides 
/usr/share/emacs/23.2/lisp/org/org-exp
/usr/share/emacs23/site-lisp/org-mode/org-install hides 
/usr/share/emacs/23.2/lisp/org/org-install
/usr/share/emacs23/site-lisp/org-mode/org-icalendar hides 
/usr/share/emacs/23.2/lisp/org/org-icalendar
/usr/share/emacs23/site-lisp/org-mode/org-remember hides 
/usr/share/emacs/23.2/lisp/org/org-remember
/usr/share/emacs23/site-lisp/flim/hmac-def hides 
/usr/share/emacs/23.2/lisp/net/hmac-def
/usr/share/emacs23/site-lisp/flim/sasl-ntlm hides 
/usr/share/emacs/23.2/lisp/net/sasl-ntlm
/usr/share/emacs23/site-lisp/flim/sasl hides /usr/share/emacs/23.2/lisp/net/sasl
/usr/share/emacs23/site-lisp/flim/sasl-digest hides 
/usr/share/emacs/23.2/lisp/net/sasl-digest
/usr/share/emacs23/site-lisp/flim/hmac-md5 hides 
/usr/share/emacs/23.2/lisp/net/hmac-md5
/usr/share/emacs23/site-lisp/flim/ntlm hides /usr/share/emacs/23.2/lisp/net/ntlm
/usr/share/emacs23/site-lisp/flim/sasl-cram hides 
/usr/share/emacs/23.2/lisp/net/sasl-cram
/usr/share/emacs23/site-lisp/wl/rfc2368 hides 
/usr/share/emacs/23.2/lisp/mail/rfc2368
/usr/share/emacs23/site-lisp/cedet-common/cedet-files hides 
/usr/share/emacs/23.2/lisp/cedet/cedet-files
/usr/share/emacs23/site-lisp/cedet-common/mode-local hides 
/usr/share/emacs/23.2/lisp/cedet/mode-local
/usr/share/emacs23/site-lisp/cedet-common/inversion hides 
/usr/share/emacs/23.2/lisp/cedet/inversion
/usr/share/emacs23/site-lisp/cedet-common/cedet hides 
/usr/share/emacs/23.2/lisp/cedet/cedet

Features:
(shadow sort mail-extr message sendmail regexp-opt ecomplete rfc822 mml
easymenu mml-sec password-cache mm-decode mm-bodies mm-encode mailcap
mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums mailabbrev nnheader
gnus-util netrc time-date mm-util mail-prsvr gmm-utils wid-edit
mailheader canlock sha1 sha1-el hex-util hashcash mail-utils emacsbug
perl-mode w3m-load org-install muse-autoloads emacs-goodies-el
emacs-goodies-custom emacs-goodies-loaddefs easy-mmode cedet-autogen
advice advice-preload autoload help-fns preview-latex tex-site
auto-loads tooltip ediff-hook vc-hooks lisp-float-type mwheel x-win
x-dnd font-setting tool-bar dnd fontset image fringe lisp-mode register
page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock
font-lock syntax facemenu font-core frame cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew
greek romanian slovak czech european ethiopic indian cyrillic chinese
case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button
minibuffer faces cus-face files text-properties overlay md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote make-network-process dbusbind system-font-setting
font-render-setting gtk x-toolkit x multi-tty emacs)