Re: [Savannah-hackers-public] Web CVS server unavailable

[Bob Proulx]

Ludovic Courtès wrote:
> This particular file takes 7 MiB and has a large history (it is updated
> daily), which perhaps makes things worse.

Oh!  I apologize that I didn't fully investigate this with the
previous exchange.  Because vcs often gets overloaded I assumed that
was the problem again.  Because when I looked it was all very happy at
that time.  That was bad of me.  Sorry.

But reading this I now realize that it is a somewhat different
problem.

  vcs:/srv/cvs/web/guix/guix# du -sh
  1.9G    .

Wow!  1.9G of web page!  That is a LOT.  Most of it in one file.

  vcs:/srv/cvs/web/guix/guix/packages# ll
  total 1267580
  -r--r--r--+ 1 civodul guix 1243780406 Apr 19 11:07 index.html,v
  -r--r--r--+ 1 civodul guix   54200311 Apr 14 01:29 issues.html,v

Oh my goodness.  That is a very large rcs file!

Looking in the file I see that the main body of the web page is all on
*one* line.  There are 29 lines in total.  Wow.  If it were newline'd
normally that would probably have made a huge reduction in the diff
sizes.

  wget -O-.index.html https://www.gnu.org/software/guix/packages/ | wc 
index.html
     29  427729 7002030

There are 293 versions checked in.  Which means that pretty much every
version is a complete copy of the file since line based diffs are
saved and there is only one line.  That isn't going to be
sustainable.  The actual root error is this one.

  vcs:/tmp/junk# cvs -d /srv/cvs/web/guix co guix/packages
  cvs checkout: Updating guix/packages
  cvs [checkout aborted]: Could not map memory to RCS archive
  /srv/cvs/web/guix/guix/packages/index.html,v: Cannot allocate memory

vcs has 6 gig of virtual memory available.  This must be exceeding a
different limit.  But this is why the web page isn't updating.  It is
hard stuck and isn't going to make any more progress until something
changes.

> Also, I’m under the impression that web page updates are queued.  For
> instance the timestamp at the bottom of the page above says “April 14”,
> whereas the commit from now succeeded (and so it should read “April
> 19”.)

Normally they are updated reasonably quickly.  But the above problem
has this stuck and is why it isn't updating at the moment.  It can't
be checked out.

Normally there is a convoluted Rube Goldberg machine that hits a URL
on the FSF machine which touches a file which then runs a script and
sees the flag files telling it to do a cvs update right now on the
page.  That usually flows through in the amount of time it takes the
machine to operate and check out the file.  (This was broken about a
month ago and fixed and therefore I believe it is operating now.)  So
normally there is no queue as such but not quite immediately.  There
are propagation delays in the mirroring.

> Regardless, if GNU packages and other projects hosted on Savannah can no
> longer update their web pages, then it’s the beginning of the end.  We
> should make sure the FSF shares this conclusion and that the issue will
> be addressed in a timely fashion, IMO.

Obviously my first feedback was wrong.  Sorry.  And I know that led to
the above which is cascade failure on the top of my misdirection.  So
I will ignore that.

But I wanted to say that Assaf Gordon has spent a significant amount
of time trying to improve the web page system on the savannah.gnu.org
side of things.  Primarily to enable use of git for web page updates.
But there are limits to what we can do on our (savannah) side of
things.  Presently his efforts are waiting for help on the fsf.org
side of things.  The improvements are stuck until more resources are
available to help there.

> Thanks for your feedback!

My feedback is that something different needs to be done for this
index.html page.  I don't want to suggest something without thinking
about it.  I might suggest something worse!  But the first thought I
had was that this needs some kind of dedicated status page.  This
shouldn't really be checked into cvs.  This looks to be some type of
real time dynamically created status page that shows the information
you want.  I would put this on a different location entirely.  Perhaps
a dedicated page?  That seems like the way it should go.  If you want
to have it displayed right where it is displayed then an <iframe> can
be used to place it there.

How is the data on that page generated?

Bob