Re: [Pan-users] pan cmd line things

[Duncan]

markballard posted on Wed, 11 Sep 2013 16:11:48 -0400 as excerpted:

> On 09/03/2013 08:19 AM, Duncan wrote:
>> markballard posted on Mon, 02 Sep 2013 15:47:14 -0400 as excerpted:
>>
>>> I'm using linux (amd64/gentoo) and pan r0.139.
>>
>> Greetings, fellow gentooer! =:^)
> 
> thanks for the response, very helpful.  I've been using linux pan gui
> since olden times (0.12/0.13 or something) when iirc it was using gtk
> something-or-other for article/header/index(?) storage/mgmt - fun stuff
> if you tried loading too many headers or got stuck w/too many in cfg and
> tried to restart pan ;)

Yes.  FWIW, I started with pan with the gnome-1 version, 0.11 something. 
0.12 was the first gtk2 series, IIRC...  And yes, pan definitely scales 
*FAR* better than it did back then, for sure.  But it still uses some of 
the same basic ideas, such as message-id for cache-file name, and still 
has a strong "traditionalist" news emphasis, with a developer and user 
base tending to favor quote/reply in that order and to view HTML posts as 
aoler at best, and possibly an attempt at security compromise of the 
reader.

> I use gnus for usenet text and nzbperl for bins.  nzbperl is great but
> lacking in a couple useful areas so I've been trying to wean myself off
> it and use pan non-gui as a replacement.
> 
> 
>> If you happen to be a coder[1], I'm sure patches would be welcomed to
>> fix the broken functionality, and even if not, having someone that's
>> actually
> 
> unfortunately I know just enough to be dangerous.  I did look over
> source code after your response but I'm afraid what I see is over my
> head.

Such is life I guess.

But I think the biggest problem with no-gui mode (other than actually 
having coders with time to actually code, a problem that has been a 
reasonably consistent one for pan between the times when someone actually 
has that time and goes to town with it!) has been that we haven't had 
anyone consistently using it, as we do GUI mode, requesting features and 
spotting and reporting regressions.  If you can be that person, and it 
looks like you may indeed fit the bill, then when development DOES 
happen, no-gui mode can take advantage of it and get its needs taken care 
of just as normal gui mode, with all the regular users.  Just that will 
be an enormous step in the right direction, no-gui-mode-wise.

However, not being a coder yourself, it's likely to take some patience, 
because as I said, pan development tends to go great guns when there's 
someone with the skills, time and interest all three, to put in, and 
stall out with even distro bugfixes having a hard time getting committed 
upstream for long periods, sometimes years, in between.  And we're in an 
in-between period now.  The last dev (Heinrich) to do a lot for pan 
really DID a lot, moving it forward quite a bit and introducing features 
like binary posting that had been on the wish list for literally over a 
decade, but while he's still interested, real life took over, and he 
simply hasn't had the same time for pan lately, so while tested bugfixes 
still get in, that's about all that's happening ATM.

And honestly, I'm not sure if pan is functional /enough/ in no-gui mode 
to maintain your interest long enough to be that no-gui tester when 
things do start happening again.  I definitely hope so, tho, because pan 
really has long been missing a user and list regular both involved enough 
to provide feedback and primarily interested in no-gui mode, to provide 
feedback for *IT*, and you're the best candidate I've seen for that in 
some time.

>>> b)  ctl-q shows in gui as how to close pan but I can't figure out how
>>> to properly close pan when using --no-gui ("q" nor ctl-q stop it, I
>>> have to ctl-c).
>>
>> Yes.  ctrl-q is a very common X-based-app "quit" keyboard shortcut -- I
>> believe the standard one for gtk/gnome apps.
>>
>> But being primarily an X-app standard keyboard shortcut, it's not for
>> non- gui mode, which is actually designed to be run "headless", as
>> part of a
> 
> what I was thinking was, when I run mplayer in non-fullscreen mode (eg)
> and hit "q" in the terminal (not the video window) it exits - I
> thought/hoped non-gui pan would receive/handle keystrokes the same way.

I think it /could/.  It's just that nobody's been interested enough to 
actually have that request and get it in front of the people able to do 
something about it when they have the time and interest in doing it, 
yet.  Hopefully you're that person. =:^)

>>> c)  I'm not finding a way to have pan not redownload part of a rar set
>>> that already exists.  for example I start an nzb d/l, have to stop
>>> pan,
> 
>> Well, that's actually due to two factors.  The first is as already
>> (sort- of) stated, that pan's no-gui mode is designed to be invoked
>> on-demand for a particular task, which it completes and then exits. 
>> The assumption
> 
> you cover a lot of useful things in your response to this item so pardon
> if it seems like I overlooked any, didn't understand some or aren't
> taking them into account.
> 
> using block accts as I do means duplicates have a cost.

Definitely.  I have a block account too, because it's the only reasonable 
way to do things for people like me that can go months without any 
activity, then get interested again and want to do some downloads while 
they have the time and motivation available.

And as I was reading your post, I was cringing inside thinking about the 
implications on my own block account...  (Plus, I remembered how 
frustrated I was all those years ago when I didn't understand the caching 
concept and ended up overwriting cached messages before I'd even looked 
at them!)

>  e.g., I wasn't
> paying attn and restarted something I'd unknowingly already d/l and
> ended up with 500M of copies of files that were already there :(  this
> could happen when killing non-gui and restarting w/o editing tasks.nzb,
> and will happen if non-gui exited normally but one inadvertently reloads
> the same nzb.

As I said, if you have a big enough cache size set and haven't actually 
deleted the messages or cleared cache, you at least avoid the redownload, 
because the actual articles will still be in cache, and both pan and nzbs 
track articles by message-id, so if the cached messages are already 
there, they won't be redownloaded, simply redecoded from cache and saved 
again.

But for that to work properly you absolutely *MUST* have a cache size 
larger than your single-complete-session download size -- and we're 
talking encoded size here too, only ~5% more (plus headers and any text-
commentary-content) than actual binary payload file size with yEnc, but 
~34% more (eight-bits-text-download-encoding-six-bits-of-binary-file, 
plus headers) with traditional UUE or MIME/base64, since it's the raw 
wire-format articles that are cached.

Which means, if practically doable for you (disks are large and space is 
cheap, but they aren't zero-cost nor storage space infinite), figure out 
how much you download in a complete session (not an interrupted one), and 
set your pan cache size accordingly, with a reasonable cushion.  Given 
that you're doing block accounts, you must already have /some/ idea of 
session size.  Then clear cache between sessions.

That's what I do, tho on a smaller scale than you may need to use.

> so I'm not entirely clear on your example of stills (where ppl may want
> the duplicates?) but wouldn't an individual bin post (or part of nzb)
> always result in a single/unique filename?  I notice with nzbperl it
> checks if the resulting article filename already exists and skips it if
> so.  pan just goes ahead and grabs another copy and would keep getting
> copies until you're out of disc space - something seems fundamentally
> wrong there.

It's not that people actually want duplicates.  It's that filenames are 
not a reliable method of determining duplicate.  If a particularly 
popular line of cameras/phones uses a consistent default filenaming, say 
snap0001.jpg, snap0002.jpg... then in a non-professional photos group 
where people don't always take care to rename to something more 
meaningful, over time it's quite likely that you'll see several different 
series with the same snapNNNN.jpg names, that are *NOT* dups and that a 
downloader may wish to keep copies of without repeatedly overwriting the 
earlier downloads.

Now I already mentioned my usual way around that, here.  I simply 
download everything that looks interesting to cache, then go do something 
else while its downloading.  Then come back and save off individual 
series to individual subdirs for which I've chosen names manually, so 
identical filenames in independent series won't conflict.  Then when I'm 
done I can clear cache and start over.

But not everybody does that, and pan's most automated modes obviously 
assume everything in a particular group will go to the same directory, as 
it has provision for automatically sorting into dirs by groupname, etc.  
Similarly, a default 10MB cache size obviously assumes direct download 
and save in a single step, not the method I use, separate download to 
cache, then go thru again when everything's already cached, and sort and 
save in an entirely separate step from the actual downloading to cache.

I'd guess the same problem exists in theory with nzbs, but probably 
doesn't occur as often in practice, because nzbs tend to be used for much 
larger files with far fewer files thus actually downloaded, and thus far 
less chance of name-collision both purely statistically and because the 
fewer/larger files tend to get more care taken on the poster's end to 
properly name them.

> thanks again for all the info/explanations duncan, I appreciate it.

=:^)



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman