Re: [Pan-users] 'Old' Pan (pan-0.14.2.91-4mdk.i586.rpm): No longer installs (Mageia)

[Duncan]

Heinrich Müller posted on Mon, 17 Dec 2012 21:28:35 +0100 as excerpted:

> I'm in the process of implementing a virtual folder for the sent
> messages.
> Will be in the next release hopefully.

That would be useful.

The thing is, there's a bit of a technical challenge there, given pan's 
message storage philosophy of using the message-id/guid (with 
substitutions where necessary) as the filesystem name.  I know I watched 
Charles struggle with that in old-pan, and that challenge is one reason  
pan's draft message storage is implemented as it is, user choosing the 
name, and why current pan doesn't have any sent message store, as well.

As you obviously know but other readers may not realize, in the cache, 
pan simply uses the the message-id, which by RFC standard must be unique 
(GUID in modern parlance, globally unique id) and is already (at minimum, 
I believe there's a bit more limitation to it as well but haven't read 
the specs in awhile) limited to printable ASCII for backward 
compatibility by the spec, as the filename.  (On POSIX standard Unix/
Linux this works quite well due to the relatively relaxed filename 
standard.  On MSWormOS, filesystem naming restrictions are worse, so more 
substitutions must be made for unusual characters.)

What you may not know if you weren't around for old-pan, for it, both the 
saved-messages and stored-messages "folders" were implemented as pseudo-
newsgroups, with articles saved to them using the same message-id-as-
filename basis.  For saved messages, this simply meant copying a message 
from the newsgroup it had been in, to the saved messages folder.  Behind 
the scenes, pan could simply copy the existing cached article file (named 
by message-id) from the cache dir to the saved messages dir (tho I don't 
remember if there was actually a separate dir or if the cache was used 
either way, making it even simpler), creating the fake overview index 
entry in the saved-messages pseudo-newsgroup as part of the process.

For sent messages, the problem was a bit more complex.  IIRC there was no 
separate messages dir for sent messages -- they simply used the cache.  
There was a double technical GOTCHA, however.  First, because the message-
id was used, that meant that pan had to assign a message-id before 
uploading the message to the server, so it /had/ a message-id to use as 
the actual article filename.  (Many news clients let the server assign a 
message-id of its own, which a server will do if the article doesn't have 
one assigned already.  Pan couldn't do that, it had to assign the ID 
before sending the message to the server.  Once set, the message-id can't 
be changed as that would in effect create an entirely new/different 
message, so servers must leave the message-id as is, when it already 
exists.)  Of course this made pan's messages much easier to trace, while 
some people prefer to remain anonymous with their postings, as much as 
possible, anyway, making this quality a big negative for some users.  
Technically, however, it simply meant that pan had to maintain bug-free 
code for create the message-id, that it could have otherwise avoided, 
letting the server do that.

A second GOTCHA was that as a result of having the message ID already in 
cache, pan would never download the message from the server, as it saw it 
already had it.  (This is the same mechanism pan uses to avoid 
redownloading cross-posts and part of what keeps multiple download 
threads from interfering with each other as well.  If the article file 
already exists in cache, downloading is simply skipped.)

This created a rather large problem from the user perspective, as unless 
they deliberately deleted the cached sent-message, they could never see 
the message as it was actually posted by the server, thus making it much 
more difficult to actually verify that the message was posted intact, at 
least on the original server -- that the server or transmission process 
didn't screwup somewhere along the line.  Quite a few people complained 
about this over the years, and I know it was irritating to me as well.


So when Charles did the rewrite into C++, he decided to change this.  Pan 
no longer had either sent-messages or saved-messages as such.  Instead, 
as I mentioned up-thread, pan avoided the need for a saved-message
folder/pseudogroup with the fact that expiry was now set per server by 
the user, so the user could (and I did) set no expiry, and simply delete 
messages they didn't want saved.  (Of course the cache size management 
would eventually delete them anyway, especially with the default 10 MB 
cache, but people like me set a huge multi-gig cache even for text-group-
only use, to avoid that as well.)

That seemed to work well for saved messages, and in fact, many users 
(including me) were very pleased, as the old mechanism, copying messages 
to a saved-messages folder/pseudogroup destroyed the context of which 
group the message was originally in, and now that context was preserved. 
=:^)

But it killed the auto-saved sent-messages concept entirely.  The (poor, 
as it was really a different feature entirely, pan previously had no way 
to save a draft message other than to send it) replacement Charles came 
up with was draft message saving and restore.  This was a great feature 
on its own, but a very poor substitute for automatically saving sent 
messages, since all too often, people didn't save a draft copy manually, 
and the server failed to post the message, making for lots of wasted 
effort recomposing lost messages!  Of course that made for a lot of 
unhappy users as well.  I know, as I lost my share of multi-hundred-line 
messages I might have spent an hour (at least) on, before I started 
saving them manually as drafts, just in case.

But, it DID allow people to actually SEE what the server got.  
Previously, if the message was corrupted in-transit, the pan-using author 
would never know unless other readers complained (and then the pan-using 
author couldn't see the damage for themselves), as the copy they saw in 
the newsgroup was the copy pan had saved off as it sent it.

But that DID allow pan to drop the whole pseudo-newsgroup thing it had 
been doing with both sent-messages and saved-messages, which meant 
Charles didn't have to recode that when he was doing the C++ rewrite.

It also means pan /could/ omit the message-id assignment code and let the 
server handle it, or better yet, make it an option, but currently, pan 
still sets the message-id before sending, for each message it sends. 


So, doing a sent-messages folder thing again would be useful.  But it's 
worth knowing the problems of the old solution, so as to ideally avoid 
them this time around.

I can think of a few ways of doing that.  One would be to still assign a 
message id and use that as the filename, but store sent messages in their 
own directory, separate from the cache.  Presumably there'd be separate 
size and expiry options for this directory as well, so it wouldn't have 
to grow indefinitely and people could choose to limit by either size or 
time-kept.  This would allow users to download their own messages from 
the server as they do now, thus allowing them to verify how the message 
was actually posted, without conflicting with the separate sent 
messages.  Additionally, using the same message-id-as-filename idea would 
avoid having to come up with a different solution, and allow users to 
track down the message by ID and post a supersedes, if desired.

(A real fancy pan GUI implementation might have a choice in the context 
menu to look for the message in sent messages and repost, as either a 
supersedes or a simple repost.  But of course that's optional.  With this 
implementation, and a small tweak to allow nixing the existing message-id 
so a replacement could be generated for the new post, a user could at 
least check the message-id of a goofed up post and use the existing open-
draft functionality to repost, doing it manually, thus avoiding all the 
new code of the fancy implementation.)

Another would be to effectively use a second message-id, presumably 
generated using the same algorithm but with an additional component, say 
-sent added to the end of the existing message-id.  Pan could then save 
and track sent messages separately, using the existing cache dir.  
Because these messages would have their own (local-only) separately 
trackable element, they'd not interfere with downloading and verification 
of one's own posts from the server, but all the features of the first 
idea could still be used, including separate expiry and size management, 
an optional nice GUI for re-editing and supersedes, etc.

Of course, an implementation not based on message-id storage is also 
possible, but I don't see any positives to it, and why do all the work of 
thinking up and debugging an entirely different implementation when we 
already have a quite well proven one.

And of course, an implementation much like that of old-pan, using 
existing message-ids and preventing a user from seeing and verifying the 
message as it appears on the server, could be done as well.  But with 
only small tweaks as suggested above, the known down aides of that 
approach can be avoided, and I'd suggest doing just that, avoiding this 
option.

(Now off to save-draft-before-sending, just in case...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman