[Pan-devel] Thinking out loud about tasks in 0.12.1

[Charles Kerr]

I've got the code written to serialize tasks to tasks.xml, and it
works well.  The xml is pretty readable IMO, and I think this will
be enough to get a cron'able Pan going sooner rather than later.
IMO we should initially point people to tasks.dtd and let them
build their own XML files by hand or with an XML editor.

The decode and download tasks still eat up a lot of memory because
they insist that all the articles in the group actually be in memory
via group_ref_articles().  I'd like to move to a point where these
tasks just take message-ids and not need to ref the Articles at all.
In particular this would make alt.binaries sessions a lot more
lightweight.

Two obstacles to having tasks take message-ids instead of Article*s:

(1) the event mechanism of setting article_set_flag (a, DOWNLOAD_QUEUED)
    or article_set_flag (a, SAVE_QUEUED) to put the little blue triangle
    in the header pane showing that this article is queued, requires
    Article*

(2) nntp.c is designed to take Article*'s.  It uses the subject and
    linecount for status-item feedback, message-id to actually get
    the message, and calls article_set_headers for "Newsgroups:",
    "Reply-To:", "Followup-To:", and "Xref:"

For the first problem, IMO these are transient queue-related states,
and that the queue could track which message-ids it has, and what their
state is.  When this tracking changes, it could fire an event that the
header pane catches, just like acache does.  A happy side-effect of this
is that it would enable the "don't-queue-an-article-if-it's-already-queued"
feature that's periodically requested.

For the second problem, we could make the status-feedback coarser
by moving the progressbar a little after each article is downloaded,
rather than each line.  We could have task-[save,bodies] remember the
subject and message-id of each article it needs to download and pass
those in to nntp.c.

We don't actually use Xref: -- we get it in XOVER -- so we would probably
be better off not looking for it in nntp_article_download_to_gstr().

Likewise Newsgroups:, Reply-To:, and Followup-To: are only used when
reading or replying to that particular article, so we should probably change
the memory/speed tradeoff for them by not keeping them in the Article*
tables but rather pumping them from acache when necessary.

cheers,
Charles