[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Savannah-cvs] [325] Mirmon,Mirroring: update information
From: |
assafgordon |
Subject: |
[Savannah-cvs] [325] Mirmon,Mirroring: update information |
Date: |
Tue, 21 Feb 2017 20:44:04 -0500 (EST) |
Revision: 325
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=325
Author: agn
Date: 2017-02-21 20:41:54 -0500 (Tue, 21 Feb 2017)
Log Message:
-----------
Mirmon,Mirroring: update information
Modified Paths:
--------------
trunk/sviki/Mirmon.mdwn
Added Paths:
-----------
trunk/sviki/Mirroring.mdwn
Modified: trunk/sviki/Mirmon.mdwn
===================================================================
--- trunk/sviki/Mirmon.mdwn 2017-02-18 03:29:49 UTC (rev 324)
+++ trunk/sviki/Mirmon.mdwn 2017-02-22 01:41:54 UTC (rev 325)
@@ -1,6 +1,10 @@
Mirmon at Savannah
==================
+NOTE: This page describe the previous implementation.
+In 2017, Savannah's infrastructure was upgraded and improved.
+See [[Mirroring]] for updated information.
+
Our [mirmon](http://www.staff.science.uu.nl/~penni101/mirmon/) instance
operates on download.savannah.gnu.org. We run [three instances of
mirmon](http://download.savannah.gnu.org/mirmon/): one each for http
Added: trunk/sviki/Mirroring.mdwn
===================================================================
--- trunk/sviki/Mirroring.mdwn (rev 0)
+++ trunk/sviki/Mirroring.mdwn 2017-02-22 01:41:54 UTC (rev 325)
@@ -0,0 +1,263 @@
+Mirroring at Savannah
+=====================
+
+NOTE: This is an updated version of the previous [[Mirmon]] page.
+
+
+Savannah offers two public-facing mirroring services:
+
+1. `ftpmirror.gnu.org` - redirects to world-wide mirrors of `ftp.gnu.org`.
+ These mirrors carry official releases of GNU pacakges.
+ `ftpmirror.gnu.org` and `gnu mirror` are used interchangeably in this
+ page.
+
+2. `download.savannah.{nongnu,gnu}.org` - redirects to world-wide mirrors of
+ savannah's [[DownloadArea]]. Typically used for non-gnu software,
+ but gnu packages can store files there as well.
+ `download` and `nongnu mirror` are used interchangeably in this page.
+
+
+Client-side overview
+--------------------
+
+A user requesting a download from `ftpmirror.gnu.org` will receive
+an HTTP/302 response redirecting to a mirror closest to the user's
+originating IP:
+
+ $ curl -o /dev/null -D /dev/stdout --silent
http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
+ HTTP/1.1 302 Found
+ Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
+ [...]
+
+
+A user requested a download from `download.savannah.gnu.org` will be
+redirected to an intermediate redirection URL, and then to the mirror:
+
+ $ curl -o /dev/null -D /dev/stdout --silent
http://download.savannah.gnu.org/releases/test-project/README
+ HTTP/1.1 302 Found
+ Location:
http://download.savannah.gnu.org/releases-redirect/test-project/README
+ [...]
+
+ $ curl -o /dev/null -D /dev/stdout --silent
http://download.savannah.gnu.org/releases-redirect/test-project/README
+ HTTP/1.1 302 Found
+ Location: http://nongnu.askapache.com//test-project/README
+ [...]
+
+The technical reason for this extra redirection is that
+`download.savannah.gnu.org` serves directory listing directly, and
+only redirects actual files. See nginx details below.
+
+
+
+Server-side redirection overview
+--------------------------------
+
+Redirection is implemented using a perl cgi script,
+called from nginx/fcgiwrap running on `download0.sv.gnu.org`.
+
+Both gnu and nongnu redirections are served from
+download0.savannah.gnu.org VM. `ftpmirror.gnu.org` is
+CNAME'd to `ftpmirror.sv.gnu.org`, which points
+to the same IP as download0 (remember savannah admin's
+only control *.sv.gnu.org DNS entries).
+
+The nginx configuration files are:
+`download0:/etc/nginx/sites-available/ftpmirror`,
+`download0:/etc/nginx/sites-available/ftpmirror-common.inc`,
+`download0:/etc/nginx/sites-available/download`,
+`download0:/etc/nginx/sites-available/download-common.inc`.
+The `*-common.inc` files contain the cgi configurations.
+The other files include them twice (once for http, once for https.
+**TODO:** consolidate them, like Bob did for `vcs0`).
+
+The CGI configuration for `gnu/ftpmirror` is:
+
+ location / {
+ fastcgi_param PATH_INFO $uri;
+ fastcgi_param MIRROR_FILE
/opt/savannah/mirror-lists/gnu-ftpmirrors.txt;
+ fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+ include fastcgi_params;
+ fastcgi_pass unix:/var/run/fcgiwrap.socket;
+ }
+
+The CGI configuration for `nongun/download` is:
+
+ # redirect to the CGI if it's a file URL
+ rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;
+
+ # directory listing if it's a directory
+ location /releases {
+ alias /srv/download;
+ fancyindex on; # Enable fancy indexes.
+ }
+
+ # actual redirection mechanism
+ location /releases-redirect {
+ fastcgi_split_path_info ^(/releases-redirect)(/?.+)$;
+ fastcgi_param PATH_INFO $fastcgi_path_info;
+ fastcgi_param MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
+ fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+ include fastcgi_params;
+ fastcgi_pass unix:/var/run/fcgiwrap.socket;
+ }
+
+The mirror list text files (`gnu-ftpmirror` and `download-mirrors.txt`)
+contain an auto-generated list of active mirrors:
+
+ # head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirror
+ http://mirror.tochlab.net/pub/gnu/ ru
+ http://mirror.cedia.org.ec/gnu/ ec
+ http://www.gutscheinrausch.de/mirror/gnu/ de
+ http://www.artfiles.org/gnu.org/ de
+ http://gnu.mirrors.linux.ro/ ro
+
+See 'mirmon' section below to learn how these files are generated.
+
+The perl script `mirror-redirect.pl` uses
+[Geo::IP](https://metacpan.org/pod/Geo::IP) and `Geo::Mirror` modules
+to read the mirror file, detect the closest region based on the user's
+IP, and return an HTTP redirect to that mirror.
+
+
+Server-side Mirmon overview
+---------------------------
+
+NOTE: Most of this section was copied from the [[Mirmon]] page
+and updated as needed.
+
+1. The authoratative lists of gnu/nongnu mirrors are managed manually.
+ For gnu/ftpmirrors, FSF admins or GNU webmasters update the lists
+ (<http://www.gnu.org/prep/ftp.html>, generated from
+ <http://www.gnu.org/prep/FTP>).
+ For nongnu/download mirrors, savannah admins update
+ [this file](http://dl.sv.gnu.org/releases/00_MIRRORS.txt)
+ (and [html version](http://dl.sv.gnu.org/releases/00_MIRRORS.html)).
+
+2. [mirmon](http://www.staff.science.uu.nl/~penni101/mirmon/) is used
+ to checks each mirror and determine how up to date it is.
+
+ We maintain 3 separate mirmon configurations:
+
+ * `gnu` - list of HTTP-only mirrors for ftpmirror.gnu.org.
+ The `gnu` list is used later by the redirection cgi script.
+ * `allgnu` - list of HTTP,FTP,RSYNC mirrors for ftpmirror.gnu.org.
+ The `allgnu` list is used only by human admins to check which
+ mirrors are up-to-date.
+ * `nongnu` - list of HTTP-only mirrors for download.savannah.gnu.org.
+ The `nongnu` list is used later by the redirection cgi script.
+
+3. The mirmon input lists (of mirrors to check) are here:
+
+ The lists are stored here:
+
+ $ ls -lhog /opt/savannah/mirrors/mirmon-lists/
+ -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt
+ -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt
+ lrwxrwxrwx 1 28 Feb 20 23:28 nongnu-mirror-list.txt ->
/srv/download/00_MIRRORS.txt
+
+ The script `/opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl`
+ fetches the gnu/ftpmirror list and saves it locally in the two versions
+ (`gnu` and `allgnu`). The `nongnu` list is symlinked to the
manually-managed
+ file.
+
+ These files are the *input* to mirmon.
+
+4. The three mirmon configuration files are:
+
+ $ ls -1 /etc/mirmon/
+ mirmon-allgnu.conf
+ mirmon-gnu.conf
+ mirmon-nongnu.conf
+
+ When `mirmon` is executed (once for each configuration),
+ it reads the corresponding list file, and generates an HTML report
+ page and a 'state' textual file:
+
+ /opt/savannah/www/mirmon/gnu/index.html
+ /opt/savannah/www/mirmon/nongnu/index.html
+ /opt/savannah/www/mirmon/allgnu/index.html
+
+ # ls -lhog /var/lib/mirmon/state-*
+ -rw-r--r-- 1 16K Feb 21 19:42 /var/lib/mirmon/state-allgnu
+ -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu
+ -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
+
+ A 'state' file contains the status for each mirror:
+
+ $ head -n5 /var/lib/mirmon/state-gnu
+ http://ca.mirror.babylon.network/gnu/ 1487646961 ok 1487651999 s
1487651999-s 1487651999
+ http://download.nust.na/pub/ftp.gnu.org/gnu/ 1487643361 ok 1487651999
s 1487651999-s 1487651999
+ http://fr.mirror.babylon.network/gnu/ 1487643361 ok 1487651999 s
1487651999-s 1487651999
+ http://ftp-gnu-org.ip-connect.vn.ua/ 1487625361 ok 1487651999 s
1487651999-s 1487651999
+ http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1487556961 ok 1487651999 s
1487651999-s 1487651999
+
+5. Two auxiliary scripts
+ `/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh`
+ read the gnu/nongnu state files, and generate a simple text file
+ containing only the active and up-to-date mirrors:
+
+ $ ls -1 /opt/savannah/mirrors/active-mirror-lists/
+ download-mirrors.txt
+ gnu-ftpmirrors.txt
+
+ These are the two files used in the nginx/cgi scripts (see above section).
+
+
+6. The 3 mirmon invocations are executed as cronjobs in
+ `/etc/cron.d/sv-mirmon`:
+
+ # Update the authoratative list of GNU ftp mirrors (mirroring
ftp.gnu.org).
+ 3 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl
--http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
+ 6 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl
--all /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
+
+ # Check the status of each mirror (and regenarate the HTML report
page).
+ # For gnu/nongnu - also update the list of active mirrors (for the
mirror CGIs).
+ 22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get
update >/var/log/mirmon/gnu 2>&1 &&
/opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh
+
+ 32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q
-get update >/var/log/mirmon/nongnu 2>&1 &&
/opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh
+
+ 42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q
-get update >/var/log/mirmon/allgnu 2>&1
+
+
+Diagnostics and Troubleshooting
+-------------------------------
+
+1. The three mirmon HTML reports are viewable here:
+ <http://download0.savannah.gnu.org/mirmon/>.
+
+2. An additional CGI script exercise the Geo::IP
+ module and the generated files:
+ <http://download0.savannah.gnu.org/mirror-check/> and
+ <http://ftpmirror0.savannah.gnu.org/mirror-check/>.
+
+ The script is `/opt/savannah/cgi-bin/mirrors/mirror-check.pl`.
+
+ The NGINX configurations are `ftpmirror-common.inc` and
+ `download-common.inc`:
+
+ location /mirror-check/ {
+ fastcgi_param GNU_MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
+ fastcgi_param NONGNU_MIRROR_FILE
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
+ fastcgi_param SCRIPT_FILENAME
/opt/savannah/cgi-bin/mirrors/mirror-check.pl;
+ include fastcgi_params;
+ fastcgi_pass unix:/var/run/fcgiwrap.socket;
+ }
+
+3. Mirmon cronjob logs are saved in `/var/log/mirmon`.
+ Current implementation saves only the most recent run.
+
+4. Mirmon state files should be generated at 22/32/42 minutes
+ of each hour (defined in the cron file):
+
+ # ls -lhog /var/lib/mirmon/
+ total 60K
+ -rw-r--r-- 1 7.3K Feb 21 20:22 state-gnu
+ -rw-r--r-- 1 3.5K Feb 21 20:32 state-nongnu
+ -rw-r--r-- 1 16K Feb 21 20:42 state-allgnu
+
+ The active mirror list files should have similar timestamp:
+
+ # ls -lhog /opt/savannah/mirrors/active-mirror-lists/
+ total 8.0K
+ -rw-r--r-- 1 869 Feb 21 20:32 download-mirrors.txt
+ -rw-r--r-- 1 2.6K Feb 21 20:22 gnu-ftpmirrors.txt
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Savannah-cvs] [325] Mirmon,Mirroring: update information,
assafgordon <=