[Savannah-cvs] [143] SavannahInternals: more information

[Assaf Gordon]

Revision: 143
          
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=143
Author:   agn
Date:     2015-02-18 00:41:49 +0000 (Wed, 18 Feb 2015)
Log Message:
-----------
SavannahInternals: more information

Modified Paths:
--------------
    trunk/sviki/SavannahInternals.mdwn

Modified: trunk/sviki/SavannahInternals.mdwn
===================================================================
--- trunk/sviki/SavannahInternals.mdwn  2015-02-16 01:45:36 UTC (rev 142)
+++ trunk/sviki/SavannahInternals.mdwn  2015-02-18 00:41:49 UTC (rev 143)
@@ -83,9 +83,8 @@
    calls a cgi script on `www.gnu.org` to request a website update.
 6. A cron job on `www.gnu.org` regularly checks for update requests,
    and `cvs updates` the requested repositories.
-7. Unlike code repositories (which use cron+database access), mailing list
-   creation is performed by `frontend` running a remote command on
-   `lists.gnu.org` using ssh.
+7. A cron job on `internal` regularly query the database for mailing list
+   updates. It then creates the mailing lists on `lists.gnu.org`.
    This is done because `lists.gnu.org` is managed by the FSF, not Savannah.
 8. Publishing a new tarball release for GNU packages is performed by an
    automated process, explained in the
@@ -295,10 +294,9 @@
   are done by the script). New mailing lists are added as new records in the
   tables. Change requests to mailing lists are done by changing the
   `status` field (see source code for possible values).
-* A cron job on `frontend` reads the changes in the `mail_group_list` and
-  executes the necessary changes on `lists.gnu.org` using ssh, then updates
-  the database with the result of the change.  
-  See 
[/backend/mail/sv_mailman.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in).
 
+* A cron job on `internal` reads the changes in the `mail_group_list` and
+  executes the necessary changes on `lists.gnu.org`. See "cron jobs on
+  internal" section below.
 
 ### MySQL Database
 
@@ -357,15 +355,15 @@
       file, ensuring it will not be overwritten. The manually added
       information is the owner name/email for some repositories. Example:
 
-          $ cat /srv/git/project-list
-          ...
-          administration.git Savannah+Hackers
-          anubis.git address@hidden
-          autoconf.git address@hidden
-          automake.git address@hidden
-          autostrap.git Sylvain+Beucler
-          coreutils.git Jim+Meyering
-          ...
+            $ cat /srv/git/project-list
+            ...
+            administration.git Savannah+Hackers
+            anubis.git address@hidden
+            autoconf.git address@hidden
+            automake.git address@hidden
+            autostrap.git Sylvain+Beucler
+            coreutils.git Jim+Meyering
+            ...
 
       These names (e.g. `Jim Meyering`) will appear
       in the *owner* field when viewing the gitweb page of the project
@@ -381,15 +379,15 @@
       mentioned project, writes an entry in `cgitrepos`. Example:
 
 
-          $ cat /srv/git/cgitrepos
-          ...
-          repo.group=GNU Core Utilities
-          repo.url=coreutils.git
-          repo.path=/srv/git/coreutils.git
-          repo.desc=GNU coreutils
-          repo.readme=README.html
-          repo.owner=Jim Meyering
-          ...
+            $ cat /srv/git/cgitrepos
+            ...
+            repo.group=GNU Core Utilities
+            repo.url=coreutils.git
+            repo.path=/srv/git/coreutils.git
+            repo.desc=GNU coreutils
+            repo.readme=README.html
+            repo.owner=Jim Meyering
+            ...
 
 * `generate_log_accum.pl` - This script updates the CVS hooks in savannah
   repositories.
@@ -402,54 +400,186 @@
       `<project>/CVSROOT/{loginfo,commitinfo}` files.
     * Example of `commitinfo` for source-code (not web) CVS repository:
 
-          $ cat /sources/pspp/CVSROOT/commitinfo
-          #<savane>
-          DEFAULT /usr/local/bin/commit_prep.pl -T pspp_70 %p
-          #</savane>
-          ALL $CVSROOT/CVSROOT/no-commits %{s}
+            $ cat /sources/pspp/CVSROOT/commitinfo
+            #<savane>
+            DEFAULT /usr/local/bin/commit_prep.pl -T pspp_70 %p
+            #</savane>
+            ALL $CVSROOT/CVSROOT/no-commits %{s}
 
     * Example of `loginfo` for source-code (not web) CVS repository:
 
-          $ cat /sources/pspp/CVSROOT/loginfo
-          #<savane>
-          DEFAULT /usr/local/bin/log_accum.pl --mail-to address@hidden 
--send-diff --file-prefix pspp_70 --config /etc/log_accum-sources.config %p 
%{sVv}
-          #</savane>
+            $ cat /sources/pspp/CVSROOT/loginfo
+            #<savane>
+            DEFAULT /usr/local/bin/log_accum.pl --mail-to address@hidden 
--send-diff --file-prefix pspp_70 --config /etc/log_accum-sources.config %p 
%{sVv}
+            #</savane>
 
     * Example of `loginfo` for web CVS repository:
 
-          $ cat /srv/cvs/web/datamash/CVSROOT/loginfo
-          #<savane>
-          ALL echo 'Triggering webpages update...'; cat > /dev/null; curl 
http://www.gnu.org/new-savannah-project/new.py -s -F type=gnu -F 
project=`basename %r`
-          #</savane>
+            $ cat /srv/cvs/web/datamash/CVSROOT/loginfo
+            #<savane>
+            ALL echo 'Triggering webpages update...'; cat > /dev/null; curl 
http://www.gnu.org/new-savannah-project/new.py -s -F type=gnu -F 
project=`basename %r`
+            #</savane>
 
 * `sv_groups` - This script creates new repositories for projects in the
   savannah database
     * reminder: when a new project is approved/created on
       savannah, or when a project administrator selects a new repository
       feature the `frontend` php code only updates the database. this
-      script picks up the changes and executes them).
+      script picks up the changes and executes them.
     * file is 
[/backend/accounts/sv_groups.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/accounts/sv_groups.in).
+    * The script queries the `groups` table, and creates the needed
+      repositories.  
+      Example: For project `XXX`, if `groups.use_git=1`, it look for
+      `vcs:/srv/git/XXX.git`. If it doesn't exist, create it with
+      `git init` and additional savannah-specific configurations (e.g.
+      permissions, hooks, xattrs).
+    * Repository initialization and configuration is delegated to the
+      relevant perl module. See
+      
[Git.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Git.pm),
+      
[Cvs.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Cvs.pm),
+      
[Bzr.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Bzr.pm),
+      
[Svn.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Svn.pm),
+      
[Hg.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Hg.pm),
+      
[Download.pm](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/lib/Savane/Download.pm).
+    * Two types of CVS repositories can be created:
+      `cvs` (Using `CvsMakeArea` in `CVS.pm`) - source code repository;
+      `webcvs` (Using `WebCvMakeAreaSavannahGNU` in `CVS.pm`) - webcvs
+      repository (content will be available on www.gnu.org/www.nongnu.org).
+    * The script also contains legacy code to create unix groups on the
+      local machine (using `/usr/sbin/groupadd`). This part is not used any
+      more, since groups and users are delegated to `nsswitch/libnss-mysql`
+      (see [[UserAuthentication]]).
+    * FIXME: What as "Area Attic" (mentioned in the code)?
+    * The script takes parameteres determining which items to create (e.g.
+      `--only-git`).
+    * On `vcs` host, the git/hg/bzr/cvs/cvsweb repositories are created.
+    * On `download` host, the download (and legacy 'arch') are created.
 
-
 ### Cron jobs on download
 
-FIXME: work-in-progress
-
 stored in `download:/etc/cron.d/sv`:
 
     */30 * * * *       root    sv_groups --cron --only-download --only-arch
 
+* `sv_groups` - This script creates new repositories for projects in the
+  savannah database. See description for `sv_groups` in the above section.
+
 ### Cron jobs on frontend
 
-FIXME: work-in-progress
+* `frontend:/etc/cron.d/sv_export` [non-functional] - contains the
+  following commands:
 
-* `frontend:/etc/cron.d/sv_export` - unused?
-* `frontend:/etc/cron.daily/sv_list_groups` - unfinished. publish
-  authoritative list of gnu/nongnu projects on savannah.
+        # XML data export
+        02 * * * *      root    sv_export --xml-path /var/cache/savane
 
-FIXME: where's the script which creates lists on `lists.gnu.org` (
-calls `sv_mailman`)?
+        # Remove old data export
+        30 1 * * *      root    sv_export_cleaner --xml-path /var/cache/savane
 
+        # Get list of Git repositories
+        3,33 * * * *    root    scp vcs.sv.gnu.org:/srv/git/cgitrepos 
/etc/savane
+
+  
[sv_export](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/export/sv_export.in)
+  is an old feature which exports project data into XML.  
+  FIXME: remove this?
+
+* `frontend:/etc/cron.daily/sv_list_groups` [unfinished] - an attempt to
+  publish an authoritative list of active projects hosted on GNU Savannah.
+  The script contains the following code:
+
+        mysql savane -B -e 'SELECT group_type.type_id, group_type.name,
+                                   unix_group_name, group_name
+                            FROM groups, group_type
+                            WHERE groups.type = group_type.type_id
+                                  AND groups.status="A"
+                            ORDER BY type;' \
+                            > /var/www/overlay/cooperation/groups.tsv
+
+  FIXME: complete this nice feature, and expose it with apache
+  configuration. Also possibly with JSON data, and more fields.
+
+### Cron jobs on internal
+
+stored in `internal:/etc/cron.d/sv`:
+
+    # Assign uidNumber and gidNumber inthe database (and rebuild cache for
+    # libnss-mysql-bg). 7211 was the latest group created by the old
+    # sv_groups cron - maybe one day we'll move everything to the 70000
+    # range. Then create homedirs and authorized_keys files.
+    */5 * * * *   root   sv_assign_uid_gid --min-gidNumber=77212
+    
+    
+    # New list creation
+    */5 * * * *     root    sv_mailman --cron
+    
+    # Database cleaning
+    */25 * * * *    root    sv_cleaner --cron
+    48 12 2 * *     root    sv_cleaner --cron --big-cleanup
+    
+    # Reminder
+    4 4 * * *       root    sv_reminder --cron
+    25 18 * * *     root    sv_reminder --cron
+
+
+* `sv_assign_uid_gid` - creates unique user ids and group ids for
+  users/projects in the savannah system.
+    * reminder: across all savannah hosts, unix user/group ids are managed
+      with nsswitch/libnss-mysql (see [[UserAuthentication]] for details).
+    * Source code:
+      
[/backend/accounts/sv_assign_uid_gid.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/accounts/sv_assign_uid_gid.in).
+    * When a new user (or project) is created on savannah, the php code
+      on `frontend` does not assign uid/gid.
+    * This script picks up new users (SQL: `user.uidNumber is NULL`) and
+      new projects (SQL: `groups.gidNumber is NULL`) and assigns them ids.
+
+* `sv_mailman` - queries the `mail_group_list` table, and creates/updates
+  the mailing lists on `lists.gnu.org`.
+    * Source code:
+      
[/backend/mail/sv_mailman.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in).
+    * This script is tightly-coupled with the php code in
+      
[/frontend/php/mail/admin/index.php](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/frontend/php/mail/admin/index.php).
+    * The `index.php` code sets the `mail_group_list.status` field to the
+      requested new status, and the `sv_mailman` script executes the
+      changes and updates the database.
+    * `sv_mailman` calls `internal:/usr/sbin/newlist`. The `newlist`
+      script is a stub which delegates the call to `lists.gnu.org` using
+      SSH. See [[MailSystem]] for more details.  
+      The gist of `internal:/usr/sbin/newlist` is:
+
+            #!/bin/bash
+            list_full_name=$1
+            admin_mail=$2
+            password=$3
+            cat <<EOF | ssh -T -i ~/.ssh/id_dsa.lists.gnu.org address@hidden
+            COMMAND=newlist
+            LIST_FULL_NAME=$list_full_name
+            ADMIN_MAIL=$admin_mail
+            PASSWORD=$password
+            EOF
+
+
+* `sv_cleaner` - cleans up left-overs from interrupted php code?
+  * source code:
+    
[/backend/misc/sv_cleaner.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/misc/sv_cleaner.in).
+  * FIXME: investigate this script.
+
+* `sv_reminder` - sends email reminders to users.
+  * source code:
+    
[/backend/misc/sv_reminder.in](http://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/misc/sv_reminder.in).
+  * The help message says:
+
+            This script will send reminder to users in some specific cases. 
+            Case 1: an item is about to start or finish, all people from the 
item 
+            notification list should be warned. 
+            Case 2: an user decided to receive a batch of open item assigned to
+            him each x days, minimum priority being > 5
+            Case 3: a project admin decided that members assigned to some item 
of 
+            should receive a batch each x days, minimum priority > 5
+            (Case 4: a project admin decided that members assigned to some item
+            whose statut/resolution havent changed for some period should 
receive
+            a batch each x days)
+
+  * FIXME: investigate this script.
+
 ### www.gnu.org update mechanism
 
 ### Mailing list configuration
@@ -467,5 +597,3 @@
 
 ### cvsweb hooks on vcs
 
-### other scripts on vcs
-