savannah-cvs
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Savannah-cvs] [325] Mirmon,Mirroring: update information


From: assafgordon
Subject: [Savannah-cvs] [325] Mirmon,Mirroring: update information
Date: Tue, 21 Feb 2017 20:44:04 -0500 (EST)

Revision: 325
          
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=325
Author:   agn
Date:     2017-02-21 20:41:54 -0500 (Tue, 21 Feb 2017)
Log Message:
-----------
Mirmon,Mirroring: update information

Modified Paths:
--------------
    trunk/sviki/Mirmon.mdwn

Added Paths:
-----------
    trunk/sviki/Mirroring.mdwn

Modified: trunk/sviki/Mirmon.mdwn
===================================================================
--- trunk/sviki/Mirmon.mdwn     2017-02-18 03:29:49 UTC (rev 324)
+++ trunk/sviki/Mirmon.mdwn     2017-02-22 01:41:54 UTC (rev 325)
@@ -1,6 +1,10 @@
 Mirmon at Savannah
 ==================
 
+NOTE: This page describe the previous implementation.
+In 2017, Savannah's infrastructure was upgraded and improved.
+See [[Mirroring]] for updated information.
+
 Our [mirmon](http://www.staff.science.uu.nl/~penni101/mirmon/) instance
 operates on download.savannah.gnu.org.  We run [three instances of
 mirmon](http://download.savannah.gnu.org/mirmon/): one each for http

Added: trunk/sviki/Mirroring.mdwn
===================================================================
--- trunk/sviki/Mirroring.mdwn                          (rev 0)
+++ trunk/sviki/Mirroring.mdwn  2017-02-22 01:41:54 UTC (rev 325)
@@ -0,0 +1,263 @@
+Mirroring at Savannah
+=====================
+
+NOTE: This is an updated version of the previous [[Mirmon]] page.
+
+
+Savannah offers two public-facing mirroring services:
+
+1.  `ftpmirror.gnu.org` - redirects to world-wide mirrors of `ftp.gnu.org`.
+    These mirrors carry official releases of GNU pacakges.
+    `ftpmirror.gnu.org` and `gnu mirror` are used interchangeably in this
+    page.
+
+2.  `download.savannah.{nongnu,gnu}.org` - redirects to world-wide mirrors of
+    savannah's [[DownloadArea]]. Typically used for non-gnu software,
+    but gnu packages can store files there as well.
+    `download` and `nongnu mirror` are used interchangeably in this page.
+
+
+Client-side overview
+--------------------
+
+A user requesting a download from `ftpmirror.gnu.org` will receive
+an HTTP/302 response redirecting to a mirror closest to the user's
+originating IP:
+
+    $ curl -o /dev/null -D /dev/stdout --silent 
http://ftpmirror.gnu.org/sed/sed-4.4.tar.xz
+    HTTP/1.1 302 Found
+    Location: http://gnu.mirrors.hoobly.com/gnu/sed/sed-4.4.tar.xz
+    [...]
+
+
+A user requested a download from `download.savannah.gnu.org` will be
+redirected to an intermediate redirection URL, and then to the mirror:
+
+    $ curl -o /dev/null -D /dev/stdout --silent 
http://download.savannah.gnu.org/releases/test-project/README
+    HTTP/1.1 302 Found
+    Location: 
http://download.savannah.gnu.org/releases-redirect/test-project/README
+    [...]
+
+    $ curl -o /dev/null -D /dev/stdout --silent 
http://download.savannah.gnu.org/releases-redirect/test-project/README
+    HTTP/1.1 302 Found
+    Location: http://nongnu.askapache.com//test-project/README
+    [...]
+
+The technical reason for this extra redirection is that
+`download.savannah.gnu.org` serves directory listing directly, and
+only redirects actual files.  See nginx details below.
+
+
+
+Server-side redirection overview
+--------------------------------
+
+Redirection is implemented using a perl cgi script,
+called from nginx/fcgiwrap running on `download0.sv.gnu.org`.
+
+Both gnu and nongnu redirections are served from
+download0.savannah.gnu.org VM. `ftpmirror.gnu.org` is
+CNAME'd to `ftpmirror.sv.gnu.org`, which points
+to the same IP as download0 (remember savannah admin's
+only control *.sv.gnu.org DNS entries).
+
+The nginx configuration files are:
+`download0:/etc/nginx/sites-available/ftpmirror`,
+`download0:/etc/nginx/sites-available/ftpmirror-common.inc`,
+`download0:/etc/nginx/sites-available/download`,
+`download0:/etc/nginx/sites-available/download-common.inc`.
+The `*-common.inc` files contain the cgi configurations.
+The other files include them twice (once for http, once for https.
+**TODO:** consolidate them, like Bob did for `vcs0`).
+
+The CGI configuration for `gnu/ftpmirror` is:
+
+    location / {
+        fastcgi_param PATH_INFO            $uri;
+        fastcgi_param MIRROR_FILE          
/opt/savannah/mirror-lists/gnu-ftpmirrors.txt;
+        fastcgi_param SCRIPT_FILENAME      
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+        include fastcgi_params;
+        fastcgi_pass unix:/var/run/fcgiwrap.socket;
+    }
+
+The CGI configuration for `nongun/download` is:
+
+    # redirect to the CGI if it's a file URL
+    rewrite ^/releases/(.*[^/])$ /releases-redirect/$1 last;
+
+    # directory listing if it's a directory
+    location /releases {
+        alias /srv/download;
+        fancyindex on;              # Enable fancy indexes.
+    }
+
+    # actual redirection mechanism
+    location /releases-redirect {
+        fastcgi_split_path_info            ^(/releases-redirect)(/?.+)$;
+        fastcgi_param PATH_INFO            $fastcgi_path_info;
+        fastcgi_param MIRROR_FILE          
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
+        fastcgi_param SCRIPT_FILENAME      
/opt/savannah/cgi-bin/mirrors/mirror-redirect.pl;
+        include fastcgi_params;
+        fastcgi_pass unix:/var/run/fcgiwrap.socket;
+    }
+
+The mirror list text files (`gnu-ftpmirror` and `download-mirrors.txt`)
+contain an auto-generated list of active mirrors:
+
+    # head -n5 /opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirror
+    http://mirror.tochlab.net/pub/gnu/ ru
+    http://mirror.cedia.org.ec/gnu/ ec
+    http://www.gutscheinrausch.de/mirror/gnu/ de
+    http://www.artfiles.org/gnu.org/ de
+    http://gnu.mirrors.linux.ro/ ro
+
+See 'mirmon' section below to learn how these files are generated.
+
+The perl script `mirror-redirect.pl` uses
+[Geo::IP](https://metacpan.org/pod/Geo::IP) and `Geo::Mirror` modules
+to read the mirror file, detect the closest region based on the user's
+IP, and return an HTTP redirect to that mirror.
+
+
+Server-side Mirmon overview
+---------------------------
+
+NOTE: Most of this section was copied from the [[Mirmon]] page
+and updated as needed.
+
+1.  The authoratative lists of gnu/nongnu mirrors are managed manually.
+    For gnu/ftpmirrors, FSF admins or GNU webmasters update the lists
+    (<http://www.gnu.org/prep/ftp.html>, generated from
+    <http://www.gnu.org/prep/FTP>).
+    For nongnu/download mirrors, savannah admins update
+    [this file](http://dl.sv.gnu.org/releases/00_MIRRORS.txt)
+    (and [html version](http://dl.sv.gnu.org/releases/00_MIRRORS.html)).
+
+2.  [mirmon](http://www.staff.science.uu.nl/~penni101/mirmon/) is used
+    to checks each mirror and determine how up to date it is.
+
+    We maintain 3 separate mirmon configurations:
+
+    * `gnu` - list of HTTP-only mirrors for ftpmirror.gnu.org.
+      The `gnu` list is used later by the redirection cgi script.
+    * `allgnu` - list of HTTP,FTP,RSYNC mirrors for ftpmirror.gnu.org.
+      The `allgnu` list is used only by human admins to check which
+      mirrors are up-to-date.
+    * `nongnu` - list of HTTP-only mirrors for download.savannah.gnu.org.
+      The `nongnu` list is used later by the redirection cgi script.
+
+3.  The mirmon input lists (of mirrors to check) are here:
+
+    The lists are stored here:
+
+        $ ls -lhog /opt/savannah/mirrors/mirmon-lists/
+        -rw-r--r-- 1 7.7K Feb 20 23:30 allgnu-mirror-list.txt
+        -rw-r--r-- 1 4.0K Feb 20 23:30 gnu-mirror-list.txt
+        lrwxrwxrwx 1   28 Feb 20 23:28 nongnu-mirror-list.txt -> 
/srv/download/00_MIRRORS.txt
+
+    The script `/opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl`
+    fetches the gnu/ftpmirror list and saves it locally in the two versions
+    (`gnu` and `allgnu`). The `nongnu` list is symlinked to the 
manually-managed
+    file.
+
+    These files are the *input* to mirmon.
+
+4.  The three mirmon configuration files are:
+
+        $ ls -1 /etc/mirmon/
+        mirmon-allgnu.conf
+        mirmon-gnu.conf
+        mirmon-nongnu.conf
+
+    When `mirmon` is executed (once for each configuration),
+    it reads the corresponding list file, and generates an HTML report
+    page and a 'state' textual file:
+
+        /opt/savannah/www/mirmon/gnu/index.html
+        /opt/savannah/www/mirmon/nongnu/index.html
+        /opt/savannah/www/mirmon/allgnu/index.html
+
+        # ls -lhog /var/lib/mirmon/state-*
+        -rw-r--r-- 1  16K Feb 21 19:42 /var/lib/mirmon/state-allgnu
+        -rw-r--r-- 1 7.3K Feb 21 20:22 /var/lib/mirmon/state-gnu
+        -rw-r--r-- 1 3.5K Feb 21 19:32 /var/lib/mirmon/state-nongnu
+
+    A 'state' file contains the status for each mirror:
+
+        $ head -n5 /var/lib/mirmon/state-gnu
+        http://ca.mirror.babylon.network/gnu/ 1487646961 ok 1487651999 s 
1487651999-s 1487651999
+        http://download.nust.na/pub/ftp.gnu.org/gnu/ 1487643361 ok 1487651999 
s 1487651999-s 1487651999
+        http://fr.mirror.babylon.network/gnu/ 1487643361 ok 1487651999 s 
1487651999-s 1487651999
+        http://ftp-gnu-org.ip-connect.vn.ua/ 1487625361 ok 1487651999 s 
1487651999-s 1487651999
+        http://ftp.acc.umu.se/mirror/gnu.org/gnu/ 1487556961 ok 1487651999 s 
1487651999-s 1487651999
+
+5.  Two auxiliary scripts
+    `/opt/savannah/mirrors/scripts/update-active-mirrors-{gnu,nongnu}.sh`
+    read the gnu/nongnu state files, and generate a simple text file
+    containing only the active and up-to-date mirrors:
+
+        $ ls -1 /opt/savannah/mirrors/active-mirror-lists/
+        download-mirrors.txt
+        gnu-ftpmirrors.txt
+
+    These are the two files used in the nginx/cgi scripts (see above section).
+
+
+6.  The 3 mirmon invocations are executed as cronjobs in
+    `/etc/cron.d/sv-mirmon`:
+
+        # Update the authoratative list of GNU ftp mirrors (mirroring 
ftp.gnu.org).
+        3 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl 
--http-only /opt/savannah/mirrors/mirmon-lists/gnu-mirror-list.txt
+        6 1 * * * root /opt/savannah/mirrors/scripts/get-gnu-mirror-list.pl 
--all       /opt/savannah/mirrors/mirmon-lists/allgnu-mirror-list.txt
+
+        # Check the status of each mirror (and regenarate the HTML report 
page).
+        # For gnu/nongnu - also update the list of active mirrors (for the 
mirror CGIs).
+        22 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-gnu.conf -q -get 
update >/var/log/mirmon/gnu 2>&1 && 
/opt/savannah/mirrors/scripts/update-active-mirrors-gnu.sh
+
+        32 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-nongnu.conf -q 
-get update >/var/log/mirmon/nongnu 2>&1 && 
/opt/savannah/mirrors/scripts/update-active-mirrors-nongnu.sh
+
+        42 * * * * root /usr/bin/mirmon -c /etc/mirmon/mirmon-allgnu.conf -q 
-get update >/var/log/mirmon/allgnu 2>&1
+
+
+Diagnostics and Troubleshooting
+-------------------------------
+
+1.  The three mirmon HTML reports are viewable here:
+    <http://download0.savannah.gnu.org/mirmon/>.
+
+2.  An additional CGI script exercise the Geo::IP
+    module and the generated files:
+    <http://download0.savannah.gnu.org/mirror-check/> and
+    <http://ftpmirror0.savannah.gnu.org/mirror-check/>.
+
+    The script is `/opt/savannah/cgi-bin/mirrors/mirror-check.pl`.
+
+    The NGINX configurations are `ftpmirror-common.inc` and
+    `download-common.inc`:
+
+        location /mirror-check/ {
+            fastcgi_param GNU_MIRROR_FILE      
/opt/savannah/mirrors/active-mirror-lists/gnu-ftpmirrors.txt;
+            fastcgi_param NONGNU_MIRROR_FILE   
/opt/savannah/mirrors/active-mirror-lists/download-mirrors.txt;
+            fastcgi_param SCRIPT_FILENAME      
/opt/savannah/cgi-bin/mirrors/mirror-check.pl;
+            include fastcgi_params;
+            fastcgi_pass unix:/var/run/fcgiwrap.socket;
+        }
+
+3.  Mirmon cronjob logs are saved in `/var/log/mirmon`.
+    Current implementation saves only the most recent run.
+
+4.  Mirmon state files should be generated at 22/32/42 minutes
+    of each hour (defined in the cron file):
+
+        # ls -lhog /var/lib/mirmon/
+        total 60K
+        -rw-r--r-- 1 7.3K Feb 21 20:22 state-gnu
+        -rw-r--r-- 1 3.5K Feb 21 20:32 state-nongnu
+        -rw-r--r-- 1  16K Feb 21 20:42 state-allgnu
+
+    The active mirror list files should have similar timestamp:
+
+        # ls -lhog /opt/savannah/mirrors/active-mirror-lists/
+        total 8.0K
+        -rw-r--r-- 1  869 Feb 21 20:32 download-mirrors.txt
+        -rw-r--r-- 1 2.6K Feb 21 20:22 gnu-ftpmirrors.txt




reply via email to

[Prev in Thread] Current Thread [Next in Thread]