Re: [Savannah-hackers-public] Accessibility to Savannah's Git repositories

[Bob Proulx]

Eli Zaretskii wrote:
> Something is wrong with access to Git repositories on Savannah via
> HTTPS: attempts to do so come back with error 502.
> 
> Could someone please look into fixing that?

The problem is that savannah systems are getting hit by a botnet.  It
is browning out the web UIs on three of the systems.  This has been
going on all weekend.  The botnet is hitting the web interface
randomly selecting every possible URL.  If you can imagine every
version of every project file in every project you will know what is
happening.

The attack started late Friday.  It is at least 10k IP addresses
strong and probably a lot bigger.  It's somewhat hard to tell the
exact size.  I know that vcs0 was hit by 45k addresses in 24 hours on
Saturday but I do not know how many of those were the botnet and how
many were just nice people like you and I clicking on the web browser.
But that seems a likely upper end.

Unfortunately we weren't previously collecting trend data on that
particular statistic for vcs0 and so I don't know what is a normal
daily rate.  Not that high by a lot however.  But at least for the
future moving forward we will have this data.  Things are running
about 30 requests per second on just vcs0 at this moment.  5/s on vcs1
and 10/s on frontend0.  And sometimes it spikes significantly higher.

We are working as best we can to try to block the attack and keep the
system limping along.  But you know how these DDoS attacks go.  If
someone wants you offline then there is really no way to stop them.

In the meantime I suggest using ssh:// protocol member access for all
of the version control backends.  Because that is not http/https it is
faring better.  Checkouts and commits should still be working.  It's
really just the web UI that is problematic.

Bob