bug#16800: 24.3; flyspell works slow on very short words at the end of big file

[Aleksey Cherepanov]

Hi, Agustin!

Wow! Your 'if' in 'while's condition is very elegant. Nice!

On Fri, Feb 28, 2014 at 12:45:45PM +0100, Agustin Martin wrote:
> On Wed, Feb 26, 2014 at 09:32:02PM +0100, Agustin Martin wrote:
> > On Mon, Feb 24, 2014 at 08:03:17PM +0400, Aleksey Cherepanov wrote:
> > > I played with different (maybe wrong) implementations of
> > > flyspell-word-search-backward and measured time against t.txt
> > > (produced by the one-liner). All implementations are attached.
> > 
> > [ ... Tons of extensive and impressive debugging ... ]
> > 
> > > We could avoid capturing at all. And it works faster as shown by 4
> > > last functions.
> > 
> > Hi, 
> > 
> > Thanks a lot for the extensive debugging and for all the suggestions. I
> > have been playing with something based in your last function, but trying
> > to get something more compact, see below current status
> [ ... ]
> > I did some efficiency test and it seemed similar to those of your efficient
> > functions. Need to check further for corner cases, bugs, etc ...
> 
> Hi, Aleksey
> 
> Please find attached my first candidate for commit. Is similar to what I
> sent before, but needed to add an explicit check for word at eob in
> `flyspell-word-search-forward'. 
> 
> Will try to have more testing before committing. Seems to work well with the
> file generated by your one-liner, even with corner cases like new
> misspellings added at bob or eob, but the wider the testing the better.

I've wrote a small fuzzer. It is in attach. To run it:
$ LANG=C emacs -Q --eval '(load-file "t2.el")'
Then C-j to start. It modifies buffer you are in.

Your -forward function gets stuck. (kbd "nd SPC and C-a") could repeat
it. my-test-forward-agustin-fixed contains fix. It incorporates
simplified word-end logic: we slip forward using flyspell-get-word,
then we check eobp. Though I did not understand why -backward does not
need a similar fix and I got the answer: my mistake with (length word)
did not allow one word to be marked as duplicate.

(if condition nil ...) could be replaced with (unless condition ...)
but I do not know what one is more readable.


(kbd "nd SPC and SPC nd C-b") fails to highlight the second "nd" as
duplicate. It is a problem with bound equal to (length word) in
-backward function. I did not check it when I wrote it.
> +                      (search-forward word (length word) t))))
                         (search-forward word (1+ (length word)) t))))

One "nd" is colored as duplicate due to -backward function after that
fix. I did not touch it yet because it is a time for a break for me.

> Hope no one will generate files with words containing something in
> OTHERCHARS.

Why?

Otherchars are not rare as of ' is there for "american" dictionary. So
even this email contains such words ("while's").

BTW quite interesting flyspell behaviour could be observed with
"met'met'and": if you jump back and forth over this word then met'met
is highlighted when you are at the beginning and met'and is
highlighted when you are at the end.

Also "met'met'and met'and" highlights both met'and as mis-spelled (the
second met'and is not marked as duplicate).


Are there any variables that could affect search like
case-fold-search? My fuzzer does not set them but users could have
them set.

Thanks!

-- 
Regards,
Aleksey Cherepanov

t2.el
Description: Text document