Re: [Savannah-hackers-public] Re: bzr + savannah.gnu.org slow again

[Jim Meyering]

Michael J. Flickinger wrote:
> On 03/06/2011 07:42 PM, Alex Fernandez wrote:
>> Hi all,
>>
>> On Mon, Feb 28, 2011 at 2:02 PM, Jim Meyering<address@hidden>  wrote:
>>
>>> Glenn Morris wrote:
>>>
>>>> At 23.30 PST on Sunday 27 Feb, it is very slow again. ~ 35 mins for a ~ 5
>>>> line commit. It was alright mid-afternoon PST.
>>>>
>>>> Can you not just disable the cgit service (it seems to still be
>>>> available) until there is a solution for this problem? Being able to
>>>> commit is more important.
>>>>
>>> Hi Glenn,
>>>
>>> I've diagnosed and fixed the problem.  git.sv is now running a new binary.
>>> The trigger was a buggy spider that was provoking an infloop in cgit.
>>> Once there were 15-20 cgit.cgi process stuck in this infloop, the
>>> system would become essentially unusable.

FYI, this was the fix for the cgit problem:

  http://hjemli.net/git/cgit/commit/?id=fc384b16fb9787380746000d3cea2d53fccc548e

There's a new version of cgit as of Saturday.
I'll install it some time this week, to replace our locally-patched one.

>> I am having problems with my CVS commits right now, it is taking many
>> minutes to update a few web pages. Apparently it is stuck at the
>> "Triggering webpages update..." phase. In fact it just crashed:
>>    IOError: [Errno 4] Interrupted system call
>> but the webpage update apparently succeeded.
>>
>> The machine cvs.savannah.nongnu.org seems to be under heavy load, with
>> lots of svn blame, viewvc.cgi and cvs processes competing for i/o.
>>
>> Let me know if I can be of any help.
>>
>> Alex.
>>
>>
> load average: 105.00, 106.65, 83.33
>
> Looks like it's under high load... investigating.

It looks fine now (6 hrs later):

   07:25:28 up 75 days,  4:44,  1 user,  load average: 2.37, 2.27, 2.79

When this happens, please run something like this

d=$(date +%F.%T);
cd ~; ps auxww>ps-$d & lsof>lsof-$d & netstat --numeric-hosts>netstat-$d

so we have some clues if/when it happens again.