Re: vc-dir operation is very slow on large git repositories in Emacs 26.1

[Bastian Beischer]

Hi,

On Mon, Jun 25, 2018 at 12:40 PM, Alex Harsanyi <address@hidden> wrote:
>
> On Sat, Jun 23, 2018 at 2:24 AM, Eli Zaretskii <address@hidden> wrote:
> >> From: Alex Harsanyi <address@hidden>
> >> Date: Fri, 22 Jun 2018 20:46:59 +0800
> >> Cc: address@hidden
> >>
> >> > Does it help to set w32-pipe-buffer-size to some large value, like
> >> > 16384, and/or decrease w32-pipe-read-delay?
> >>
> >> If I set w32-pipe-read-delay to 0 the speed improvement is
> >> significant.  On top of that, there is an additional speed
> >> improvement if I set w32-pipe-buffer-size to 16384
> >
> > Can you show numbers for these speedups, and compare the times with
> > Emacs 25.  It would also be useful if you show times with different
> > values of these, so that the optimal values could be chosen.
> >
> > I'd be also interested in the timings with the larger repository.
>
> I tested various values for `w32-pipe-read-delay` on my work git repository.
> The original timings that I made were against a work tree that had several
> modified files and I used a stop watch to time the update.  These tests were
> run on a large (51678 tracked files) but clean working directory, and I
> modified the source files to time the commands using `current-time`, so I got
> more precise timings.
>
>     $ git ls-files . | wc -l
>     51678
>
>     $ git status
>     On branch ah_tmp
>     nothing to commit, working directory clean
>
> Emacs 26.1 runs the following commands as part of a `vc-dir` command, or an
> update of the `*vc-dir*` buffer (Emacs 25.2 runs the same commands, except it
> does not run the third one)
>
>     * git --no-pager update-index --refresh .
>     * git --no-pager diff-index --relative -z -M HEAD -- .
>     * git --no-pager ls-files -z -c -s -- .
>     * git --no-pager ls-files -z -o --directory --no-empty-directory
> --exclude-standard -- .
>
> On a clean working directory, only "git --no-pager ls-files -z -c -s -- ."
> produces any output, but produces 6.3 Mb of it.  The command itself runs fast:
>
>     $ git --no-pager ls-files -z -c -s -- . | wc
>           0  184463 6689336
>
>     $ time git --no-pager ls-files -z -c -s -- . > /dev/null
>
>     real    0m0.619s
>     user    0m0.000s
>     sys     0m0.015s
>
> I tried several values for `w32-pipe-read-delay` to see how long it takes to
> run these commands in Emacs 26.1:
>
> `w32-pipe-read-delay` is set to 50 milliseconds (default for Emacs 26.1)
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.26 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.34 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 83.70 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.64 seconds
>
> `w32-pipe-read-delay` set to 25 milliseconds
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.26 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.34 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 42.77 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.67 seconds
>
> `w32-pipe-read-delay` set to 12 milliseconds
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.27 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.35 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 21.57 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.65 seconds
>
> `w32-pipe-read-delay` set to 0 milliseconds (no delay)
>
>     Running git --no-pager update-index --refresh . in background...
> done 0.31 seconds
>     Running git --no-pager diff-index --relative -z -M HEAD -- . in
> background... done 0.39 seconds
>     Running git --no-pager ls-files -z -c -s -- . in background...
> done 1.07 seconds
>     Running git --no-pager ls-files -z -o --directory
> --no-empty-directory --exclude-standard -- . in background... done
> 0.69 seconds
>
> So, with a 50 millisecond delay, it takes 83.7 seconds to read the command
> output and this is reduced to 1.07 seconds when the delay is set to 0.

>From these timings it appears that you have been using a 4096 byte
buffer. The logic is as follows:

Assume that all the delay in the 83.7 seconds is due to 50ms waits,
one such wait whenever the buffer is full. This is based on the fact
that you observe a ~proportional slowdown when increasing the
read-delay. Then this means that the buffer was full 1674 times, which
means that the buffer size is 6.5 MB / 1674 = 4071 - approx. 4096
bytes. So the predicted approximate delay for a message of size S is

D = (S / B) * W

where B is the buffer size (4096) and W is the w32-pipe-read-delay
(0.05 s). Based on that logic it appears that setting
w32-pipe-buffer-size to 16384 should give you a ~four times smaller
total delay - especially when w32-pipe-read-delay is 50 ms, for
example. Unless the OS refuses to provide you with such a buffer size?

>
>
> Alex.
>

Cheers
Bastian