Re: [Pan-users] Pan/0.142 Header shown twice?

[Duncan]

Duncan posted on Mon, 09 Oct 2017 02:58:17 +0000 as excerpted:

> Dave posted on Sun, 08 Oct 2017 23:46:08 +0000 as excerpted:
> 
>> About the most complicated thing I do these days
>> apart from a bit of bash scripting is edit the Pan source code to
>> increase the number of threads to 30 per server whenever a FreeBSD port
>> comes along.
> 
> As you may know but it's worth repeating for others that may be reading
> in any case, editing source code isn't absolutely essential for that
> tweak.

> [T]he non-code workaround is to edit the config file, in this case
> servers.xml in the pan config dir (~/.pan2 by default[).]

> Meanwhile, I've one more point to make on the server connections topic,
> but I'll make it in a followup post, as it could potentially lead to its
> own subthread...

So here's that followup...

Are you sure you actually need 30 connections?  In my experience, there 
are only three (rather narrow) cases in which more than, say, 8-10 
connections, and often, any more than 4, is actually useful, and one of 
those three cases won't count here.

IOW, more connections doesn't always mean better throughput.  In fact, in 
practice more connections often means either worse throughput or more CPU 
and memory usage to get the same thruput, due to all those extra fetch 
threads trying to use the same already maxed out pipe.

The three exceptions are:

1) Users where the server caps the per-connection bandwidth but not the 
total bandwidth, so the more connections you can use the more bandwidth 
you get.

But providers with per-connection bandwidth caps generally allow only a 
handful of connections anyway; usually in the single digits and often 
only 2-4, only very rarely 20+ connections, because they're using a per-
connection bandwidth limit along with a total connections limit, as a 
substitute for what they /really/ intend, a per-user bandwidth limit, 
because the per-connection limit is easier or less resources intensive to 
track in their server software.

Given that you mentioned a 30-connection limit, this wouldn't seem to 
apply to you.  20+ connections allowed tends to be much more common where 
they're encouraging you to use more bandwidth, either to encourage tier 
upgrades on per-month plans, or to encourage you to eat thru your block 
plan as fast as possible so you'll need to renew it.

2) Those with poor quality internet connections in general.  In this 
case, the individual connections will be throttled to something well 
below the nominal pipe bandwidth, both that of the user's internet 
connection and that allowed by the server, due to TCP's packet-loss 
throttling mechanisms.

If this is you, OUCH, I feel for you!  Been there, and it SUCKS!  While 
upping the number of news server connections may help to some degree for 
news, it doesn't help for general web browsing, media streaming (which 
can be simply unworkable, period, unless you can download it to cache to 
watch later... which of course is something you might be using the binary 
newsgroups for in place of real-time streaming), any sort of interactive 
use, etc.

Due to the degree to which such poor internet situations suck in general, 
most people eventually either give up or switch to something else, if 
they possibly can.  But 20-30 connections may indeed help somewhat for 
news, in the mean time.  This one's the most legit reason, but as I said, 
if it applies, I sure feel for you!


And the one that won't apply in this case, but can be legit where it 
does...

3) Multiple different users and/or devices connecting to the same NSP 
account at once.  A good example I know is the guy who had pan 
downloading big binaries (TV programs, many from overseas) to his media 
server with some of his connections, a couple others used by pan on his 
personal computer for text groups, and a few others used by his wife/gf 
on her machine, keeping up with her embroidery or whatever it was (cross-
stitch, some such...) groups.

Obviously this makes good use of an NSP's 20+ connection limit, but 
doesn't require a whole lot of connections per individual pan instance.  
In fact, the GNKSA-max four connections per server may well be plenty, 
even for that binary-downloader media machine, if neither of the first 
two apply as well.


Meanwhile, every connection has its own overhead, TCP-connection related, 
CPU-thread context switches, and multiple write threads accessing storage 
at the same time thereby heavily fragmenting all files compared to fewer 
download threads at once.  (Of course this latter cost will apply mostly 
to conventional "spinning-rust", not so much to SSDs, which should handle 
up to gigabit pipes at least, with ease.)  If you can get the same thruput 
with fewer connections, it'll mean lower CPU usage, less disk thrashing 
if you're still on spinning rust, and less connection overhead as well.  
So fewer connections tends to be better, as long as you're not throttling 
thruput, which you shouldn't be under normal circumstances.

In fact, here, at least some years ago, I found I could max my local 
internet pipe connection with only two connections!  Of course my local 
internet pipe is bigger now (I just upgraded to 100 megabit, there's 300 
available but I can't justify it yet, and gigabit available in some areas 
of town but not yet to me), I'm on ssds so the storage IO shouldn't be a 
bottleneck, and my middle-aged AMD bulldozer-1 fx6100 6-core moderately 
overclocked to 4 GHz is better than back then as well, but I'd still be 
extremely surprised if 2-4 connections couldn't max things out just as 
well as 8 or 20 or 50, and above say 8 connections, I'd expect CPU-
overhead on the 6-core to have an increasing effect.

So what I'm saying is if you've not actually checked and can justify 
double-digit download connections, you may well want to do some checking, 
because it's actually quite likely 30 connections is doing you more harm 
than good.  OTOH if you're running a 16-core-plus monster with a gigabit 
internet pipe and storing to ssds, then perhaps all those threads /do/ 
increase your thruput, as they might if you have a really poor internet 
connection and are seeing tcp-throttling due to lost packets on 
individual connections.  But other than that, if you haven't checked, I'd 
suggest doing so.  I certainly changed my mind when I saw two connections 
maxing out my pipe just as well as many more did.  (Tho I did see a 
reasonable boost with two over just one.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman