[Pan-users] Re: One group suddenly ignored

[Duncan]

Glen Walpert posted on Fri, 08 Apr 2011 10:33:33 -0400 as excerpted:

> I have been using Pan 0.132 on Ubuntu 8.04 for a year with no problems
> until a few days ago when all posts to sci.electronics.design started
> being ignored ("get headers in subscribed groups" lists over 100
> messages but they all disappear when the group is selected, no headers
> for the last 3 days even when no filters are selected so they are not
> just marked as read.)
> 
> All of my other subscribed groups have no problems.  If it is an issue
> with my news server (news.astraweb.com) it does not affect an old copy
> of Agent on XP, which still reads new messages on
> sci.electronics.design.  I tried temporarily removing .pan2/Score with
> no effect (not a bad scoring rule) and reproduced the problem on a clean
> install of Pan 0.133 under Ubuntu 9.04 with no scoring rules.  What else
> should I look for?

First, note that pan 0.132 and previous had a security vulnerability that 
was fixed in 0.133.  pan is obscure enough it's unlikely to be a problem 
unless someone's specifically targeting you, but you really want to 
upgrade that 0.132 instance so you're no longer exposed.

It's also worth noting that 0.134 is out now, with various mostly bugfix 
patches that have been floating around for awhile.  You may wish to 
consider upgrading to it, as it's possible the issue you are seeing is one 
of those bugs (see below for details).  But for sure you want at least 
0.133, due to that security issue fix!

As to your problem, there are three possibilities here.  Well, four 
actually.  I added this top (0) one later, on top, since it's easiest to 
deal with and get out of the way.

Zeroeth, easy to test but too vaguely defined to know a fix immediately, 
report the results for further investigation.  What happens if you use the 
get headers... dialog, and select get ALL headers?  It's usual that pan 
might fill in the occasional missed post here or there, and they may or 
may not appear read, but does get all headers get the new ones that get 
headers in subscribed groups missed, or not?  If it does, report that 
behavior and perhaps we can figure out what's happening.  If not, well, we 
know it doesn't help, now.  The reasoning here is that the get all headers 
function uses somewhat different methods to request them than the other 
options do, and it can at times get stuff that isn't picked up using the 
normal fetch new headers functionality.  The problem in that case may well 
be a peculiarity of the server behavior, or a pan bug, or... but knowing 
one way or the other is more data on it than we have now.

First, easiest to fix, but probably not /your/ problem (unless you copies 
over the pan data sans scorefile from the old install to the clean 
install), and simply listed here for completeness, pan's read-message 
tracking depends on the server keeping its per-group message sequence 
numbers in order.  If the server starts its numbering over, as it might do 
if it had a storage problem for the group and restored from a backup, or 
if you switch servers by just giving pan a new server address for an 
existing server config, instead of creating a new server config for the 
new server, the new message sequence numbers may appear to pan to be for 
old messages it long since expired, instead of the new ones they actually 
are.

The fix for that is to either create a fresh server config so pan starts 
with a fresh idea of what the server message sequence numbers should be, 
OR, with pan not running naturally, to manually text-edit the appropriate 
newsrc file for that server (for multi-server people, the servers.xml file 
tracks which server uses which newsrc file), deleting pan's tracking of 
read-messages for that group, or simply delete that newsrc file entirely, 
if losing that info for all groups for that server isn't a problem.

Second, the problem alluded to above, fixed by an upgrade to 0.134.  The 
problem here is that in huge groups, typically binary groups with tens of 
thousands of individual daily articles (a full ISO binary post may be 
split over thousands of individual articles for just that one post!), the 
message sequence numbers exceed the maximum number (a bit under 4.3 
billion) that can be recorded with a 32-bit number!  32-bit pan especially 
(I'm not sure about 64-bit pan, BTW not /just/ pan, it was a problem for a 
number of news clients) used to use a 32-bit int for tracking message 
sequence numbers.  

The patch alluded to above switched pan to using a 64-bit number for the 
same thing.  That's not going to run out in the foreseeable future!  But 
that patch didn't make it into 0.133, only 0.134.  Thus, the fix in this 
case is to upgrade to 0.134.  However, a sci.* group doesn't normally 
allow binaries (does it?) and as such, is quite unlikely to have surpassed 
the 32-bit limit of some 4 billion plus posts, in which case this patch 
won't solve the problem you are seeing.

Third... not so good news.  There has been a longstanding but quite 
obscure bug in pan for quite some time, related in some way to cross-
posts.  The bug is obscure enough not to affect most folks, and I'd be 
unsure it was even there, but for the fact that I have it myself!  
Unfortunately, it's obscure enough and complicated enough nobody has been 
able to trace down the problem, and it remains unfixed.

What I know about the problem so far is that it is somehow triggered by 
crossposts to multiple groups.  I /think/ it only happens when one group 
involved in the cross-post gets ahead of another, perhaps you don't visit 
or update the behind group for some days, or perhaps you're just adding it 
as a new subscription after seeing sufficient cross-post activity to 
become interested in the other group (my case here).  Somehow, something 
about the cross-posts gets pan mixed up, with the result being, 
apparently, a case of the first possibility above -- pan somehow records 
way high message sequence numbers for the one group, perhaps recording the 
numbers for the first group cross-posted to, to the second instead of 
using the lower numbers of the second group.  As such, it then believes 
any new messages it sees are in fact years old and refuses to download 
them!

If you are seeing this rather obscure behavior, I'm sad for you, but glad, 
in that we now have another data-point to compare against mine, and maybe 
some progress can finally be made on this thing!

Two questions should help prove this the case or not:

1) Does the problem group get cross-posts from another group you regularly 
visit, or may have visited in the past?  (If no such cross-posts are 
involved, it's unlikely you're experiencing this same bug.)

2) If you start a clean pan instance (say by moving the ~/.pan2 directory 
elsewhere, when pan's not running of course, so it has to start a new 
one), setup the server, and ONLY visit the problem group, WITHOUT visiting 
anything else first, so the ONLY data it has is from that problem group, 
the group should load just fine.  However, start trying to bring the saved 
data from the other groups back in and the problem group may go 
unresponsive again.  The existing posts will remain (until they expire 
anyway), but pan will refuse to update the group any further.  (This is 
what I see here.  Again, if with a completely clean pan instance, with the 
problem group the FIRST you visit so no history or possibility at all for 
the cross-posted messages to interfere since the other group hasn't been 
visited yet, if it does NOT load just fine, you're seeing a different 
problem.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman