[Savannah-hackers-public] Diagnosing Problematic mirrors

[Assaf Gordon]

Hi,

This is in relation to:
 "fxt-0.3.4 didn't get propagated on all mirrors"
 https://savannah.gnu.org/support/?109408
 https://lists.gnu.org/archive/html/savannah-hackers/2017-11/msg00004.html

Symptom:
Requesting this URL:
  http://download.savannah.nongnu.org/releases/fkt/fxt-0.3.4.tar.gz

Redirects to some mirrors where the file is missing, e.g:
   http://mirror6.layerjet.com/nongnu/fkt/fxt-0.3.4.tar.gz

I don't have a solution yet, but here are some steps to diagnose where 
the problem is:


Step 1: Check time stamps
=========================
On the primary GNU server, this is the non-gnu time stamp:

  $ curl https://download-mirror.savannah.gnu.org/releases/00_TIME.txt
  1509739201
  Fri Nov  3 20:00:01 UTC 2017

NOTE:
If you request the timestamp from "download.savannah.gnu.org",
you'll be redirected to a mirror... so download it from 
"download-mirror" (which serves files directly from the gnu/savannah 
server).



Step 2: Check timestamps on mirrors
===================================

Now let's check the time stamps on the mirrors.
The propagation of this file (00_TIME.txt) is used by savannah's mirmon
to check the validate of the mirrors.


Download the (non-gnu) mirror list:

  wget https://download-mirror.savannah.gnu.org/releases/00_MIRRORS.txt

And check the time stamp on each:

  $ for i in $(awk '/http/ { print $3}' 00_MIRRORS.txt) ; do \
      A=${i#*//}; B=${A%%/*}; \
      printf "%-40s" "$B" ; \
      curl -s $i/00_TIME.txt | tail -n1 ; \
    done
  mirror.csclub.uwaterloo.ca              Fri Nov  3 16:00:01 UTC 2017
  babyname.tips                           Thu Nov  2 21:30:01 UTC 2017
  mirror.netcologne.de                    Fri Nov  3 15:30:01 UTC 2017
  ftp.cc.uoc.gr                           Thu Nov  2 21:00:01 UTC 2017
  mirror.infotronik.hu                    Fri Nov  3 10:45:01 UTC 2017
  mirror1.hostiran.ir                     Fri Nov  3 14:30:01 UTC 2017
  download-mirror.savannah.gnu.org        Fri Nov  3 20:15:01 UTC 2017
  nongnu.askapache.com                    Fri Nov  3 20:00:01 UTC 2017
  savannah.c3sl.ufpr.br                   Fri Nov  3 18:45:01 UTC 2017
  mirror.lihnidos.org                     Fri Nov  3 13:00:01 UTC 2017
  mirrors.up.pt                           Fri Nov  3 19:45:01 UTC 2017
  mirrors.up.pt                           Fri Nov  3 19:45:01 UTC 2017
  nongnu.uib.no                           Fri Nov  3 08:30:01 UTC 2017
  mirror.rackdc.com                       Thu Nov  2 22:00:01 UTC 2017
  ftp.igh.cnrs.fr                         Fri Nov  3 19:30:01 UTC 2017
  savannah.spinellicreations.com          Fri Nov  3 04:30:01 UTC 2017
  gnu.mirrors.pair.com                    Fri Nov  3 08:45:01 UTC 2017
  www.mirrorservice.org                   Fri Nov  3 19:15:01 UTC 2017
  savannah-nongnu-org.ip-connect.vn.ua    Fri Nov  3 14:45:01 UTC 2017
  mirror.cedia.org.ec                     Fri Nov  3 18:15:01 UTC 2017
  ftp.yzu.edu.tw                          Fri Nov  3 13:15:01 UTC 2017
  mirror6.layerjet.com                    Fri Nov  3 15:30:01 UTC 2017
  mirror.easyname.at                      Fri Nov  3 01:00:01 UTC 2017
  mirror2.klaus-uwe.me                    Fri Nov  3 20:00:01 UTC 2017
  ftp.acc.umu.se                          Fri Nov  3 05:00:01 UTC 2017
  mirror.freedif.org                      </body></html>
  www.namesdir.com                        Thu Nov  2 20:30:01 UTC 2017


Seems like most mirrors have a recent time stamp, including the 
offending "mirror6.layerjet.com".
This is why savannah's mirmon indicates they are up to date.

(mirror.freedif.org is broken, and it is detected as such on
the mirmon page, http://download.savannah.gnu.org/mirmon/savannah/#vn ).



Step 3: Check the missing file
==============================

  $ for i in $(awk '/http/ { print $3}' 00_MIRRORS.txt) ; do \
       A=${i#*//}; B=${A%%/*}; \
       printf "%-40s" "$B" ; \
       curl -I -s $i/fkt/fxt-0.3.4.tar.gz | head -n1 ; \
   done
  mirror.csclub.uwaterloo.ca              HTTP/1.1 200 OK
  babyname.tips                           HTTP/1.1 200 OK
  mirror.netcologne.de                    HTTP/1.1 200 OK
  ftp.cc.uoc.gr                           HTTP/1.1 200 OK
  mirror.infotronik.hu                    HTTP/1.1 200 OK
  mirror1.hostiran.ir                     HTTP/1.1 200 OK
  download-mirror.savannah.gnu.org        HTTP/1.1 200 OK
  nongnu.askapache.com                    HTTP/1.1 200 OK
  savannah.c3sl.ufpr.br                   HTTP/1.1 200 OK
  mirror.lihnidos.org                     HTTP/1.1 200 OK
  mirrors.up.pt                           HTTP/1.1 200 OK
  mirrors.up.pt                           HTTP/1.1 200 OK
  nongnu.uib.no                           HTTP/1.1 200 OK
  mirror.rackdc.com                       HTTP/1.1 200 OK
  ftp.igh.cnrs.fr                         HTTP/1.1 200 OK
  savannah.spinellicreations.com          HTTP/1.1 200 OK
  gnu.mirrors.pair.com                    HTTP/1.1 200 OK
  www.mirrorservice.org                   HTTP/1.1 200 OK
  savannah-nongnu-org.ip-connect.vn.ua    HTTP/1.1 200 OK
  mirror.cedia.org.ec                     HTTP/1.1 200 OK
  ftp.yzu.edu.tw                          HTTP/1.1 200 OK
  mirror6.layerjet.com                    HTTP/1.1 404 Not Found
  mirror.easyname.at                      HTTP/1.1 200 OK
  mirror2.klaus-uwe.me                    HTTP/1.1 200 OK
  ftp.acc.umu.se                          HTTP/1.1 200 OK
  mirror.freedif.org                      HTTP/1.1 404 Not Found
  www.namesdir.com                        HTTP/1.1 200 OK

So it seems that even though the timestamp file was sync'd,
some files are not (at least on "mirror6.layerjet.com".

The file "/fkt/fxt-0.3.4.tar.gz" was uploaded on 03-Aug-2017.

We cant check also with a more recent file (this sed
file was uploaded by me on 29-Oct-2017):

  $ for i in $(awk '/http/ { print $3}' 00_MIRRORS.txt) ; do\
         A=${i#*//}; B=${A%%/*}; \
         printf "%-40s" "$B" ; \
         curl -I -s $i/sed/sed-4.4.89-490b.tar.xz | head -n1 ; \
    done
  mirror.csclub.uwaterloo.ca              HTTP/1.1 200 OK
  babyname.tips                           HTTP/1.1 200 OK
  mirror.netcologne.de                    HTTP/1.1 200 OK
  ftp.cc.uoc.gr                           HTTP/1.1 200 OK
  mirror.infotronik.hu                    HTTP/1.1 200 OK
  mirror1.hostiran.ir                     HTTP/1.1 200 OK
  download-mirror.savannah.gnu.org        HTTP/1.1 200 OK
  nongnu.askapache.com                    HTTP/1.1 200 OK
  savannah.c3sl.ufpr.br                   HTTP/1.1 200 OK
  mirror.lihnidos.org                     HTTP/1.1 200 OK
  mirrors.up.pt                           HTTP/1.1 200 OK
  mirrors.up.pt                           HTTP/1.1 200 OK
  nongnu.uib.no                           HTTP/1.1 200 OK
  mirror.rackdc.com                       HTTP/1.1 200 OK
  ftp.igh.cnrs.fr                         HTTP/1.1 200 OK
  savannah.spinellicreations.com          HTTP/1.1 200 OK
  gnu.mirrors.pair.com                    HTTP/1.1 200 OK
  www.mirrorservice.org                   HTTP/1.1 200 OK
  savannah-nongnu-org.ip-connect.vn.ua    HTTP/1.1 200 OK
  mirror.cedia.org.ec                     HTTP/1.1 200 OK
  ftp.yzu.edu.tw                          HTTP/1.1 200 OK
  mirror6.layerjet.com                    HTTP/1.1 404 Not Found
  mirror.easyname.at                      HTTP/1.1 200 OK
  mirror2.klaus-uwe.me                    HTTP/1.1 200 OK
  ftp.acc.umu.se                          HTTP/1.1 200 OK
  mirror.freedif.org                      HTTP/1.1 404 Not Found
  www.namesdir.com                        HTTP/1.1 200 OK






Step 4: mirroring source
========================

The official mirroring instructions are here:
   https://www.gnu.org/server/mirror.html

The recommended mirroring command is:
(Scroll down to "Mirroring nongnu releases from Savannah" section).

    rsync -rltpHS --delete-excluded \
        rsync://dl.sv.gnu.org/releases/ /your/dir

The problematic file is certainly there:

  $ rsync -rltpHS --delete-excluded  \
         rsync://dl.sv.gnu.org/releases/fkt/ | grep "0\.3"
  -rw-rw-r--      1,088,432 2014/08/18 06:06:38 fxt-0.3.0.tar.gz
  -rw-rw-r--      1,095,022 2015/03/06 02:37:13 fxt-0.3.1.tar.gz
  -rw-rw-r--      1,121,530 2016/10/17 09:37:05 fxt-0.3.2.tar.gz
  -rw-rw-r--      1,122,946 2017/03/22 11:38:06 fxt-0.3.3.tar.gz
  -rw-rw-r--      1,123,143 2017/08/03 09:00:29 fxt-0.3.4.tar.gz

So if the mirror's operator are running this command as-is,
the should be getting the file, and it should "just work".

================

One thing I notice is that this specific mirror ("mirror6.layerjet.com" 
from france) was recently finicky, as indicated by the mirmon stats:
  http://download.savannah.gnu.org/mirmon/savannah/#fr

Perhaps they had some intermittent problems,
and their synchronization is broken?


We can contact them and alert them to this issue,
or temporarily drop them from the list.


What do you think ?

regards,
 - assaf


P.S.
In the examples above I checked on http mirrors,
but I repeated it with ftp mirrors as well.
Only "mirror6.layerjet.com" seems to be missing this file.