Re: [Pan-users] suggestions for large groups

[Mike Erickson]

* jef (address@hidden) wrote:
> Erik wrote:
> >>I was wondering if anyone could give some tips for using Pan with large
> >>binary groups. I have a gig of memory in my machine, and Pan's cache is
> >>set to a gig as well. However, loading a newsgroup with about 2 million
> >>headers ends up causing Pan to start thrashing swap space about 2/3 of the
> >>way through loading the group, or downloading that many new headers. The 
> >>machine slows down to a crawl and the rest of the (down)load takes 
> >>excessively long.
> >
> >This is because you have 1 gig of cache. Lower it to a reasonable
> >amount and try again. 10 MB here and no high memory usage, swapping,
> >etc, even on machines with 128-256MB of RAM.
> 
>       Only 10MB here, and I see the same behavior on groups this large. I 
> mentioned this problem a while back, and Charles asked for a copy of the 
> group's header file so that he could look into it.
> 
>       I haven't heard of any outcome from this, but the file i sent, for 
> alt.binaries.multimedia, is 350MB. Sorting, scoring, etc on a file that 
> big is bound to put a bit of stress on things - but I'm hoping that  the 
> Pan crew can work some magic as they have in the past.

I'm guessing* that this is one of the reasons the SQL backend is on
the todo list. I also have big bog-downs on sorting and marking-read
large groups. 

>       One of the things I noted in a previous post was that Pan used to do 
> this on groups with only 100,000 headers, but a tweak was made that 
> improved performance dramatically at the time. I'm hoping that they are 
> able to find a similar tweak for even larger groups.  (And I just got 
> word recently that Giganews is increasing binary retention, so I'm 
> hoping they can work it out :)

Do you remember what this tweak was?

* if this isn't the case, what are the benefits of moving to an SQL
backend

mike