bug#32280: 26.1; FLYSPELL-BUFFER sometimes misbehaves for some input in a large enough buffer

[Eli Zaretskii]

> From: Artem Boldarev <artem.boldarev@gmail.com>
> Date: Mon, 30 Jul 2018 21:12:36 +0300
> 
> > I'd suggest you to try lines below
> >
> > [A-Za-zАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЬЫЪЭЮЯабвгдеёжзийклмнопрстуфхцчшщьыъэюя]
> > [^A-Za-zАБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЬЫЪЭЮЯабвгдеёжзийклмнопрстуфхцчшщьыъэюя]
> >
> > with the latin chars A-Za-z added. ¿Does it work?
> 
> I have tried to do as you suggested. The result is the same as in my 
> previous letter.

And now I understand why.  The problem is not with comparing the
length of the misspelled word, the problem is with this part of
flyspell-external-point-words:

              ;; Iterate on string search until string is found as word,
              ;; not as substring.
              (while keep
                (if (search-forward word
                                    flyspell-large-region-end t)
                    (let* ((found-list
                            (save-excursion
                              ;; Move back into the match
                              ;; so flyspell-get-word will find it.
                              (forward-char -1)
                              (flyspell-get-word)))  <<<<<<<<<<<<<<<<<<<<<<
                           (found (car found-list))
                           (found-length (length found))
                           (misspell-length (length word)))

When the misspelled word doesn't match CASECHARS, the call to
flyspell-get-word will find an entirely different word than the one
which was originally found as misspelled: it will find the first word
before point that matches CASECHARS.  In your case, since the
misspelled words were in English, flyspell-get-word will find the
first Cyrillic word before point.  From there on, the logic of the
code in flyspell-external-point-words completely breaks down, and
yields results that are more-or-less random.

IOW, the assumption of the current logic in
flyspell-external-point-words is that the misspelled word is from the
same language that is supported by the current dictionary, and in your
case this assumption is false.  This is why the problem disappeared as
soon as you added Latin alphabetic characters to CASECHARS.

So please try this patch for flyspell.el, it should fix your problem
with the original setup of ru_RU (it also fixes an unrelated wrong
assumption which goes back to the days when the spell-checking program
could only be either Ispell or Aspell):

diff --git a/lisp/textmodes/flyspell.el b/lisp/textmodes/flyspell.el
index 5726bd8..4d7a189 100644
--- a/lisp/textmodes/flyspell.el
+++ b/lisp/textmodes/flyspell.el
@@ -1420,10 +1420,20 @@ flyspell-external-point-words
 The list of incorrect words should be in `flyspell-external-ispell-buffer'.
 \(We finish by killing that buffer and setting the variable to nil.)
 The buffer to mark them in is `flyspell-large-region-buffer'."
-  (let (words-not-found
-       (ispell-otherchars (ispell-get-otherchars))
-       (buffer-scan-pos flyspell-large-region-beg)
-       case-fold-search)
+  (let* (words-not-found
+         (flyspell-casechars (flyspell-get-casechars))
+         (ispell-otherchars (ispell-get-otherchars))
+         (ispell-many-otherchars-p (ispell-get-many-otherchars-p))
+         (word-chars (concat flyspell-casechars
+                             "+\\("
+                             (if (not (string= "" ispell-otherchars))
+                                 (concat ispell-otherchars "?"))
+                             flyspell-casechars
+                             "+\\)"
+                             (if ispell-many-otherchars-p
+                                 "*" "?")))
+         (buffer-scan-pos flyspell-large-region-beg)
+         case-fold-search)
     (with-current-buffer flyspell-external-ispell-buffer
       (goto-char (point-min))
       ;; Loop over incorrect words, in the order they were reported,
@@ -1453,11 +1463,18 @@ flyspell-external-point-words
                              ;; Move back into the match
                              ;; so flyspell-get-word will find it.
                              (forward-char -1)
-                             (flyspell-get-word)))
+                              ;; Is this a word that matches the
+                              ;; current dictionary?
+                              (if (looking-at word-chars)
+                                 (flyspell-get-word))))
                           (found (car found-list))
                           (found-length (length found))
                           (misspell-length (length word)))
                      (when (or
+                             ;; Misspelled word is not from the
+                             ;; language supported by the current
+                             ;; dictionary.
+                             (null found)
                             ;; Size matches, we really found it.
                             (= found-length misspell-length)
                             ;; Matches as part of a boundary-char separated
@@ -1479,13 +1496,21 @@ flyspell-external-point-words
                             ;; backslash) and none of the previous
                             ;; conditions match.
                             (and (not ispell-really-aspell)
+                                  (not ispell-really-hunspell)
+                                  (not ispell-really-enchant)
                                  (save-excursion
                                    (goto-char (- (nth 1 found-list) 1))
                                    (if (looking-at "[\\]" )
                                        t
                                      nil))))
                        (setq keep nil)
-                       (flyspell-word nil t)
+                        ;; Don't try spell-checking words whose
+                        ;; characters don't match CASECHARS, because
+                        ;; flyspell-word will then consider as
+                        ;; misspelling the preceding word that matches
+                        ;; CASECHARS.
+                        (or (null found)
+                           (flyspell-word nil t))
                        ;; Search for next misspelled word will begin from
                        ;; end of last validated match.
                        (setq buffer-scan-pos (point))))