Re: Performance issue w/ `cl-loop`s `collect...into`

[Tianxiang Xiong]

Here's a second, cleaner attempt that separates the `cl--loop-handle-accum` function into two functions, one to deal with lists and one to deal w/ non-lists.

The tail-tracking optimizing is also applied to `append(ing)` and `nconc(ing)`.

On Sun, Apr 8, 2018 at 4:29 PM, Tianxiang Xiong <address@hidden> wrote:

One thing I don't understand is the common 

(push `(progn (setq ...) t) cl--loop-body)

pattern found in the code. I'm not sure why the `(progn ... t)` is necessary. If anyone could explain that I'd add it as a comment.

On Sun, Apr 8, 2018 at 2:13 PM, Stefan Monnier <address@hidden> wrote:

> Avoid O(n^2) nconc-ing by keeping track of tail of collection.

I took a quick look at your patch, and it looks pretty good.
See comments below.


        Stefan


>       ((memq word '(collect collecting))
> -      (let ((what (pop cl--loop-args))
> -         (var (cl--loop-handle-accum nil 'nreverse)))
> -     (if (eq var cl--loop-accum-var)
> -         (push `(progn (push ,what ,var) t) cl--loop-body)
> -       (push `(progn
> -                   (setq ,var (nconc ,var (list ,what)))
> -                   t)
> -                cl--loop-body))))
> +      (let ((what (pop cl--loop-args)))
> +        (cl-multiple-value-bind (var var-tail)

`cl-multiple-value-bind` is the "destructor" corresponding to the
`cl-values` "constructor".   Since your code doesn't use `cl-values` it
should not use `cl-multiple-value-bind` either (you probably meant to
use cl-destructuring-bind instead).

> +            (cl--loop-handle-accum nil 'nreverse)
> +          (if (eq var cl--loop-accum-var)
> +              (push `(progn (push ,what ,var) t) cl--loop-body)
> +            (push `(progn
> +                     (if (null ,var-tail)
> +                         (setq ,var (list ,what) ,var-tail (last ,var))
> +                       (setq ,var-tail (setcdr ,var-tail (list ,what))))
> +                     t)
> +                  cl--loop-body)))))

The cl-loop macro's code lacks comments.  Could you take advantage of
"being there" to try and add comments?  E.g. in the above code I see
that depending on (eq var cl--loop-accum-var) we end up accumulating in
the from or in the back.  Could you add a comments explaining why and
mentioning where we correct this discrepancy?

> +      (let ((what (pop cl--loop-args)))
> +     (cl-destructuring-bind (var) (cl--loop-handle-accum nil 'nreverse)
> +          (push `(progn
> +                   (setq ,var
> +                         ,(if (eq var cl--loop-accum-var)
> +                              `(nconc
> +                                (,(if (memq word '(nconc nconcing))
> +                                      #'nreverse #'reverse)
> +                                 ,what)
> +                                ,var)
> +                            `(,(if (memq word '(nconc nconcing))
> +                                   #'nconc #'append)
> +                              ,var ,what)))
> +                   t)
> +                cl--loop-body))))

In the `nconc` case (when (eq var cl--loop-accum-var) is nil) we could
also use the `var-tail` to speed up the `nconc`.

Also, to avoid the N² behavior for the `append` case, maybe we
could/should make it use `copy-sequence`, i.e.

    `(nconc ,var-tail (copy-sequence ,what))

> -(defun cl--loop-handle-accum (def &optional func) ; uses loop-*
> -  (if (eq (car cl--loop-args) 'into)
> +(defun cl--loop-handle-accum (def &optional func type) ; uses loop-*
> +  (setq type (or type 'list))

Please add a docstring explaining whatever you managed to understand of
this code, and describing also what this new arg `type` does.

0002-Optimize-collect.into.patch
Description: Text Data