bug#47711: 27.1; Deferred highlighting support in `completion-all-completions', `vertico--all-completions`

[Daniel Mendler]

Emacs is lacking a possibility to defer the completion highlighting when
computing completions via `completion-all-completions'. This feature is
important for the performance of completion UIs when the set of all
completions is much larger than the set of completions which are
displayed.

The Vertico package defers highlighting by modifying the
`completion*-hilit-*' function with advices.

(declare-function orderless-highlight-matches "ext:orderless")
(defun vertico--all-completions (&rest args)
  "Compute all completions for ARGS with deferred highlighting."
  (cl-letf* ((orig-pcm (symbol-function 
#'completion-pcm--hilit-commonality))
             (orig-flex (symbol-function 
#'completion-flex-all-completions))
             ((symbol-function #'completion-flex-all-completions)
              (lambda (&rest args)
                ;; Unfortunately for flex we have to undo the deferred 
highlighting, since flex uses
                ;; the completion-score for sorting, which is applied 
during highlighting.
                (cl-letf (((symbol-function 
#'completion-pcm--hilit-commonality) orig-pcm))
                  (apply orig-flex args))))
             ;; Defer the following highlighting functions
             (hl #'identity)
             ((symbol-function #'completion-hilit-commonality)
              (lambda (cands prefix &optional base)
                (setq hl (lambda (x) (nconc 
(completion-hilit-commonality x prefix base) nil)))
                (and cands (nconc cands base))))
             ((symbol-function #'completion-pcm--hilit-commonality)
              (lambda (pattern cands)
                (setq hl (lambda (x) (completion-pcm--hilit-commonality 
pattern x)))
                cands))
             ((symbol-function #'orderless-highlight-matches)
              (lambda (pattern cands)
                (setq hl (lambda (x) (orderless-highlight-matches 
pattern x)))
                cands)))
    (cons (apply #'completion-all-completions args) hl)))

This function `vertico--all-completions` returns the list of completions 
and a highlighting
function which can then be used to highlight the completions on the fly.
It is a prototype of how some improved functionality in Emacs could look 
like.

(completion-all-completions STRING TABLE PRED POINT &optional METADATA 
DEFER-HL)

or

(completion-all-completions-defer-hl STRING TABLE PRED POINT &optional 
METADATA)

If DEFER-HL=t, then the function returns the completions and a
highlighting function. One may consider returning a triple of base,
completions and highlighting functions. Internally the completion styles
should be adapted such that they support the deferred highlighting.

It could be that this feature becomes less needed with the introduction
of gccemacs in the future.