bug#14820: 24.3; elisp manual: How to write good idle timer worker functions?

[Phil Sainty]

Documentation and/or possible enhancement needed.

Sorry, this is a wee bit long; the short version is that I'd like to
see the elisp manual provide a canonical solution to the problems
raised (but not resolved) in (elisp) Idle Timers in the paragraph
beginning "Do not write an idle timer function containing a loop"...

The long version is:


I'm trying to do some work with idle timers, but this seems like one
of the trickier aspects of Emacs to work with, and I'm finding it hard
to determine the correct approach. In particular I'm wishing that the
documentation at "(elisp) Idle Timers" provided some robust example
code which could safely be used as a template for this kind of thing.

I feel that improved documentation would probably help a lot of people.
(I would be happy to draft up some changes to the documentation myself
if only I knew for sure what that example code should look like.)

Frustratingly, the documentation actually tells me what NOT to do, but
does not tell me what I should do instead.


Here's the use case:

There is a substantial amount of work to perform 'in the background',
taking (in total) in excess of 10 seconds to complete, and also
needing to be run on a semi-regular basis, so making the user wait
each time is out of the question. I wish to perform the work in idle
time, as quickly as possible, but without disrupting the user.

The work can be broken down into a queue of small items, each of which
takes a fraction of a second to complete.

When my idle timer triggers, I would like Emacs to start processing
the queue, and continue to process the queue without pausing unless
Emacs ceases to be idle (or should be dealing with something else, at
any rate). It's this last part where I've been struggling with the
documentation.

Essentially I want to iterate over the queue, and after processing
each item I want to check to see if Emacs is waiting to do something
else. If not I would like it to continue processing immediately, not
introducing any unnecessary pauses. If there is some kind of input
or other necessary activity pending, I need to break out of my loop
and wait for Emacs to become idle again before resuming.


At present the documentation says:

> Do not write an idle timer function containing a loop which does a
> certain amount of processing each time around, and exits when
> `(input-pending-p)' is non-`nil'.  This approach seems very natural
> but has two problems:
>
>   * It blocks out all process output (since Emacs accepts process
>     output only while waiting).
>
>   * It blocks out any idle timers that ought to run during that time.

So that helpfully warns me about the wrong approach for my use case,
but doesn't say what I should do instead (the remainder of that info
node discusses intentionally introducing a delay into the processing,
which seems like a different topic -- I don't wish to introduce any
delays if I can avoid it).

The quote above indicates that `input-pending-p' is not sufficient to
detect all kinds of pending activity, but it doesn't suggest other
options. My current impression is that the `sit-for' function seems
like the way to allow Emacs check for other pending activity, and that
(sit-for 0 t) looks like a way to do so without adding unnecessary
delays or activity.

I've tried this in conjunction with `with-timeout' and it does enable
me to make Emacs break out of a loop after a certain duration, which
suggests to me that this resolves the first of the documentation's
stated problems -- blocking other idle timers (given that
`with-timeout' depends upon timers).

So with the `sit-for' call added into the mix, is it then enough to
check `input-pending-p'? Will this combination also deal with the
second stated problem of blocking process output? Or is this still
insufficient in some cases?


In other words, will something like the following be a robust
approach in general?

(defvar my-queue nil "Queue of items to process")
(defvar my-idle-duration 2 "Required idle time before starting")

(defvar my-timer nil)
(defvar my-resume-timer nil)

;; after populating the queue, initialise processing with:
(setq my-timer (run-with-idle-timer
                my-idle-duration t 'my-process-queue))

(defun my-process-queue ()
  "Process the queue until interrupted."
  (while (and my-queue (not (input-pending-p)))
    (my-process-queue-item (pop my-queue))
    (sit-for 0 t))

  (if my-queue
      (let ((idle (current-idle-time)))
        (when idle
          ;; Schedule a one-off resume timer.
          (when (timerp my-resume-timer)
            (cancel-timer my-resume-timer))
          (setq my-resume-timer
                (run-with-idle-timer
                 (time-add idle (seconds-to-time my-idle-duration))
                 nil 'my-process-queue))))
    ;; Queue is empty; cancel the repeating timer.
    (cancel-timer my-timer)
    (when (timerp my-resume-timer)
      (cancel-timer my-resume-timer))))


I also don't know if the sit-for duration of zero would prevent Emacs
from doing things that a larger number would enable? And if so, what
is the best way to ensure that Emacs will be able to deal with any
other processes, while not also causing undue delays in processing the
queue items?

Basically I don't know enough about how Emacs interacts with processes
to be able to answer these kinds of questions myself, and it would be
great if the manual provided enough detail that I could be confident
about making Emacs perform arbitrary work in idle time without
introducing any potential problems.


Perhaps there's a good case to be made for introducing some new
macro into the API to provide a simple standard way of implementing
this kind of idle-time worker function?

There's certainly enough boiler-plate in the code I've come up with
that some kind of wrapper would seem warranted, and I would suspect
that this is true even if I'm barking up the wrong tree with my
approach.


thanks,
-Phil