Sorry for the late reply - just started going through forgotten emails.
>> autorevert ends up calling vc-git-find-file-hook multiple times.
> This sounds like a problem in itself.
> Or is it really because the file gets modified multiple times?
I think it is caused by each edit. At least the symptom is that after so much idle time,
vc-git-find-file-hook gets run. Now that I'm looking for it, I don't actually see
vc-git-find-file-hook is added to any hooks, so something seems off. It may be some
kind of interaction with Magit that sets this up.