Re: A project-files implementation for Git projects

[Dmitry Gutov]

Hi Tassilo,

On 18.09.2019 20:15, Tassilo Horn wrote:

> Well, ok.  I've now played with an interface
>
>    (vc-call-backend (vc-responsible-backend dir)
>                    'list-files
>                    dir
>                    include-unregistered
>                    extra-includes)

Not sure we should use the vc-call-backend route when only 1 or 2 
backends will work well enough, but OK, it's not essential.

> where extra-includes works in addition to the standard VC ignore rules
> (.gitignore, .hgignore).  Or do you want to override the VC-internal
> rules?

I'm afraid it might not work as well if we try to treat all (modified) 
ignores the same. In my understanding, the speed with which Git lists 
files to a large extent stems from not having to apply the ignores to 
the already-registered files. Someone should benchmark this, but I think 
if we use the "negative pathspec" approach mentioned below for all 
ignores together, it might slow down file listing by an order of 
magnitude or several.

> At least for Git and Hg, I came up with reasonable implementations:
>
> --8<---------------cut here---------------start------------->8---
> (defun vc-git-list-files (&optional dir
>                                      include-unregistered
>                                      extra-ignores)
>    (let ((default-directory (or dir default-directory))
>          (args '("-z")))
>      (when include-unregistered
>        (setq args (nconc args '("-c" "-o" "--exclude-standard"))))
>      (when extra-ignores
>        (setq args (nconc args
>                          (mapcan
>                           (lambda (i)
>                             (list "--exclude" i))
>                           (copy-list extra-ignores)))))
>      (mapcar
>       #'expand-file-name
>       (cl-remove-if
>        #'string-empty-p
>        (split-string
>         (apply #'vc-git--run-command-string nil "ls-files" args)
>         "\0")))))
>
> (defun vc-hg-list-files (&optional dir
>                                     include-unregistered
>                                     extra-ignores)
>    (let ((default-directory (or dir default-directory))
>          args
>          files)
>      (when include-unregistered
>        (setq args (nconc args '("--all"))))
>      (when extra-ignores
>        (setq args (nconc args
>                          (mapcan
>                           (lambda (i)
>                             (list "--exclude" i))
>                           (copy-list extra-ignores)))))
>      (with-temp-buffer
>        (apply #'vc-hg-command t 0 "."
>               "status" args)
>        (goto-char (point-min))
>        (while (re-search-forward "^[?C]\s+\\(.*\\)$" nil t)
>          (setq files (cons (expand-file-name (match-string 1))
>                            files))))
>      (nreverse files)))

Terrific, thank you! How is Hg's performance with this approach? Does 
adding a few ignores (like 5 or 10) slow down the output measurably?

BTW, can Hg support extra whitelist entries as well?

> --8<---------------cut here---------------end--------------->8---
>
> There's a semantic difference between Git and Hg in the treatment of
> extra-ignores.  With Git, the extra-ignores do not rule out committed
> files (i.e., they are only effective for untracked files) while for Hg,
> they also rule out committed files.  I think the Hg semantics are
> probably better

Better and important, IMO.

> but I don't see how to change the Git version so that it
> acts the same way (except by re-filtering in lisp, of course), do you?

Previously suggested:

https://stackoverflow.com/questions/36753573/how-do-i-exclude-files-from-git-ls-files/53083343#53083343

That means converting all extra-ignores into negative pathspec strings.

> I haven't looked at the other backends.  I guess bzr will probably be
> doable, too.  However, for SVN, there's no way to list unregistered
> files.  A correct (but horribly slow) default implementation should also
> be doable.

Yeah, I wonder if we should treat this as a VC operation. On the other 
hand, the fallback implementation could just as well use 'find'.