savannah-cvs
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Savannah-cvs] [226] FrontEndSetup, FrontEndDevelopmentSite: new pages


From: assafgordon
Subject: [Savannah-cvs] [226] FrontEndSetup, FrontEndDevelopmentSite: new pages
Date: Wed, 21 Sep 2016 20:29:11 +0000 (UTC)

Revision: 226
          
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=226
Author:   agn
Date:     2016-09-21 20:29:08 +0000 (Wed, 21 Sep 2016)
Log Message:
-----------
FrontEndSetup,FrontEndDevelopmentSite: new pages

Modified Paths:
--------------
    trunk/sviki/SavannahHosts.mdwn

Added Paths:
-----------
    trunk/sviki/FrontEndDevelopmentSite.mdwn
    trunk/sviki/FrontEndSetup.mdwn

Added: trunk/sviki/FrontEndDevelopmentSite.mdwn
===================================================================
--- trunk/sviki/FrontEndDevelopmentSite.mdwn                            (rev 0)
+++ trunk/sviki/FrontEndDevelopmentSite.mdwn    2016-09-21 20:29:08 UTC (rev 
226)
@@ -0,0 +1,308 @@
+FrontEnd Development Sites
+==========================
+
+With the new servers we aim to provide development copies of the frontend's PHP
+code, enabling savannah admins/volunteers to develop features and fix bugs more
+quickly.
+
+Development websites will have URLs such as
+<https://foobar.frontend0.sv.gnu.org>.
+The PHP code will be hosted in each developer's home directory,
+(e.g. `frontend0:/home/foobar/projects/savane/`).
+
+See also: [[FrontEndSetup]], [[SavannahHosts]].
+
+
+Step 1: DNS Configuration
+-------------------------
+
+Add a new DNS CNAME record, currently done on
+`internal.sv.gnu.org` (not on `internal0` - it is not yet
+the active DNS server).
+
+See general DNS instructions in [[DNS]].
+
+Add the following line to `internal:/etc/bind/master/savannah.footer`:
+
+    foobar.frontend0   CNAME   frontend0
+
+Update `/etc/bind/master/savannah.header` with new serial number,
+and reload with `rndc reload`.
+
+Use a hostname alone (e.g. `frontend0`) - do not use fully qualified
+domain name (e.g. `frontend0.savannah.gnu.org`).  The DNS setup
+ensures that a hostname will be included in all four possibilities:
+
+    foobar.frontend0.savannah.gnu.org
+    foobar.frontend0.savannah.nongnu.org
+    foobar.frontend0.sv.gnu.org
+    foobar.frontend0.sv.nongnu.org
+
+
+
+Step 2: MySQL database copy
+---------------------------
+
+Create a dedicated copy of 'savane' mysql database, to ensure changes
+on the development site do not affect the production site.
+
+Create a new MySQL database and dedicated user (Example in
+`internal0:/root/agn/create-savane-dev-db.sh`. This script
+sets a password for the user). The result is:
+
+    mysql> select user,host from mysql.user where user='foobar';
+    +---------+----------------------------+
+    | user    | host                       |
+    +---------+----------------------------+
+    | foobar  | frontend.savannah.gnu.org  |
+    | foobar  | frontend0.savannah.gnu.org |
+    | foobar  | localhost                  |
+    +---------+----------------------------+
+    3 rows in set (0.01 sec)
+
+Create a script to copy the existing data to the development database
+copy. See example in `internal0:/root/agn/copy-savane-db-to-agn.sh`.
+This process takes about 1 minute - safe to re-run whenever an update
+is needed.  All existing data in the development DB will be deleted.
+
+NOTE: An improved version is being discussed where only a smaller
+subset of the database is copied, to save disk space on the server.
+
+Test connection from `frontend0`:
+
+    ## Ensure it requires a password:
+    $ mysql -h internal0.sv.gnu.org -u foobar savane_foobar
+    ERROR 1045 (28000): Access denied for user 
'foobar'@'frontend0.savannah.gnu.org' (using password: NO)
+
+    ## With the password:
+    $ mysql -h internal0.sv.gnu.org -u foobar -p savane_foobar
+    Enter password: ******
+    mysql>
+
+    ## Ensure the dev user (foobar) can not access the real database:
+    $ mysql -h internal0.sv.gnu.org -u foobar -p savane
+    Enter password:
+    ERROR 1044 (42000): Access denied for user 
'foobar'@'frontend0.savannah.gnu.org' to database 'savane'
+
+
+
+
+Step 3: PHP frontend code
+-------------------------
+
+Clone the savane repository
+
+    cd /home/foobar
+    mkdir projects ; cd projects
+    git clone git://git.sv.gnu.org/administration/savane.git
+
+The current PHP code (the same code running on the existing/old 'frontend')
+is few old PHP constructs. Few patches are required to make it run on modern
+PHP installations. The updated repository is available in
+`frontend0:/home/agn/projects/savane`, or can be downloaded here:
+`wget -O sv-local-run.patch.xz 
'https://savannah.gnu.org/support/download.php?file_id=32220'`
+
+FIXME: push these to `savane.git` repository (if not in 'master', then at least
+in a separate branch).
+
+Inside the dev. copy of savane, configure the
+copy for local installation (be sure to read below for warnings and
+notes about this 'installation'):
+
+    cd savane
+    ./bootstrap
+    ./configure --prefix=$HOME/savane-usr
+    make
+    make install
+
+Notes:
+
+1.  The `make` step builds the Savane perl modules and scripts
+    (called 'backend' - somewhat of a misnomer for the perl scripts
+    running on 'frontend').
+
+2.  The 'make' step also prepares some images in
+    `<savane>/frontend/php/images` from `*.orig.png` which are in the
+    git repository
+    (e.g. `<savane>/frontend/php/images/Savane.theme/mist/trash.png`
+    generated from
+    `<savane>/frontend/php/images/Savane.theme/mist/trash.orig.png`.)
+    See `<savane>/frontend/php/images/Makefile.am` for the commands.
+
+    Without this step, some image URLs will result in HTTP-404 errors.
+
+3.  The 'make install' will create the following directory structure:
+
+        savane-usr
+        |-- bin
+        |-- etc
+        |   `-- savane
+        |       `-- content
+        |           |-- account
+        |           |-- admin
+        |           |-- arch
+        |           |-- bzr
+        |           |-- cvs
+        |           |-- git
+        |           |-- hg
+        |           |-- mail
+        |           |-- my
+        |           |-- people
+        |           |-- predefined_mails
+        |           |-- project
+        |           |-- register
+        |           `-- svn
+        |-- lib
+        |   `-- perl
+        |       `-- 5.18.2
+        |           `-- auto
+        `-- share
+            |-- locale
+                |   |-- ca
+        ...
+
+4. `make install` does *not* install savane's PHP frontend code. The
+   PHP code is served directory from the git repository directory. All
+   `make install` does is to prepare the backend perl scripts and
+   the content files. When using it for development (as opposed
+   to the production website) it can be helpful to know which
+   perl scripts are installed/needed.
+
+5. The installed content directory (`$HOME/savane-usr/content/etc/content` in
+   the example above) is *not* used for the development website. See below
+   for manual configuration (which is just copied from the real website's
+   /etc directory). But for future development, it could be useful
+   to know which content files are installed/needed.
+
+
+
+Step 4: Apache configuraion
+---------------------------
+
+Create a separate apache configuration file:
+`frontend0:/etc/apache2/sites-available/sv-foobar-dev.conf`
+A symlink should exist in:
+`frontend0:/etc/apache2/sites-enabled/sv-foobar-dev.conf`
+(See example in `/etc/apache2/sites-available/sv-agn-dev.conf`.)
+
+### Hostnames
+
+The apache config file defines the following server names
+(see also DNS section above):
+
+    ServerName  agn.frontend0.savannah.gnu.org
+    ServerAlias agn.frontend0.sv.gnu.org
+    ServerAlias agn.frontend0.savannah.nongnu.org
+    ServerAlias agn.frontend0.sv.nongnu.org
+
+
+
+### SSL Certificates
+
+The apache configuration file includes an `alias` to certbot's
+".well-known" directory to allow Let's-Encrypt SSL certificate
+generation. The following URLs will resolve to
+`frontend0:/home/certbot/www/.well-known`:
+
+    http://foobar.frontend0.sv.gnu.org/.well-known/foo
+    http://foobar.frontend0.sv.nongnu.org/.well-known/foo
+    http://foobar.frontend0.savannah.gnu.org/.well-known/foo
+    http://foobar.frontend0.savannah.nongnu.org/.well-known/foo
+
+FIXME: Explain Bob's non-root certbot setup.
+
+
+
+### Document root
+
+The PHP code is served from the savane git repository directory:
+
+    DocumentRoot /home/foobar/projects/savane/frontend/php
+
+
+
+### Savane's PHP configuration
+
+In this apache config file, an environment variable `SAVANE_CONF` should
+point to the development site's configuration directory
+(e.g. `frontend0:/home/foobar/projects/savane-etc`).
+
+    SetEnv SAVANE_CONF /home/foobar/projects/savane-etc/
+
+
+
+### Password-protection
+
+This is a development website, not meant for public consumption.
+Add a simple password, so the website is available only to
+the developer (e.g. when trying developing new features).
+
+    <Location />
+        # Require all granted
+        AuthType Basic
+        AuthName "FooBar DEV"
+        AuthUserFile /etc/apache2/sv-foobar-dev-passwd
+        Require valid-user
+    </Location>
+
+
+
+PHP db/user/pw configuration
+----------------------------
+
+The PHP code reads the configuration from a `.savane.conf.php` file,
+in a directory defined by `SAVANE_CONF` in the apache configuration
+file (see above). Example: `frontend0:/home/foobar/projects/savane-etc`.
+
+The files are based on the production configuration directory in
+`frontend0:/etc/savane`.
+
+The files contain the db/user/pw for the development website:
+
+    $ cd /home/foobar/projects/savane-etc
+    $ grep '_db' .savane.conf.php
+    $sys_dbhost="208.118.235.78";
+    $sys_dbname="savane_foo";
+    $sys_dbuser="foobar";
+    $sys_dbpasswd=rtrim(file_get_contents(dirname(__FILE__).'/savane.pass'));
+
+    $ cat savane.pass
+    XXXXXX
+
+    $ cat savane.ini
+    [database]
+    HOST=208.118.235.78
+    USER=foobar
+    PASSWORD=XXXXXXXX
+    NAME=savane_foobar
+
+Other relevant settings for the development site:
+
+    $ cd /home/foobar/projects/savane-etc
+    $ cat .savane.conf.php
+    $sys_brother_domain="foobar.frontend0.savannah.nongnu.org";
+    $sys_incdir="/home/foobar/projects/savane-etc/content/gnu-content";
+    $sys_name="Savannah-FOOBAR-DEV";
+
+The PHP code runs from apache as user `www-data`.
+The configuration directory must allow `www-data` to read the files.
+
+    $ cd /home/foobar/projects/savane-etc
+    $ chgrp www-data .savane.conf.php savane.pass
+
+The outcome should look like:
+
+    $ cd /home/foobar/projects/savane-etc
+    $ ls -la
+    total 168
+    drwxr-xr-x 5 foobar foobar        4096 Sep  5 22:00 .
+    drwxrwxr-x 5 foobar foobar        4096 Jul 27 23:30 ..
+    -rwxr-x--- 1 foobar www-data      1616 Sep  5 21:58 .savane.conf.php
+    -rw-r--r-- 1 foobar foobar      123540 Jul 27 23:30 cgitrepos
+    drwxr-xr-x 5 foobar foobar        4096 Jul 27 23:30 content
+    drwxr-xr-x 2 foobar foobar        4096 Jul 27 23:30 nongnu-conf
+    -rwxr-x--- 1 foobar foobar        8533 Sep  5 22:00 savane.conf.pl
+    -rw-r----- 1 foobar foobar          90 Sep  5 21:59 savane.ini
+    -rw-r----- 1 foobar www-data        25 Sep  5 21:58 savane.pass
+    drwxr-xr-x 2 foobar foobar        4096 Jul 27 23:30 test-conf
+

Added: trunk/sviki/FrontEndSetup.mdwn
===================================================================
--- trunk/sviki/FrontEndSetup.mdwn                              (rev 0)
+++ trunk/sviki/FrontEndSetup.mdwn      2016-09-21 20:29:08 UTC (rev 226)
@@ -0,0 +1,112 @@
+Savannah Frontend Setup
+=======================
+
+This page describes the configuration needed for savannah's
+new frontend web server and PHP code.
+
+In 2016, the FSF allocated new hardware for savannah's servers,
+kicking off the server migration efforts. The existing server
+(which hosts <http://savannah.gnu.org>) is called `frontend`
+(`frontend.savannah.gnu.org`). The new server is currently
+called `frontend0` (<https://frontend0.savannah.gnu.org>).
+
+Further reading about savannah's server:
+
+* [[SavannahArchitecture]] - overview of the current ('old') setup
+  (i.e. vcs, mgt, frontend, internal, vcs, download).
+* [[SavannahServices]] - services/daemons available on savannah's hosts.
+* [[SavannahInternals]] - Savannah's inner-workings.
+* [[SavannahHosts]] - Configuration of the new servers
+  (i.e. mgt0, vcs0, frontend0, internal0, download0).
+
+Further reading about savannah's frontend:
+
+* [[RunningSavaneLocally]] - Running savannah's php code locally.
+* [[SavaneInABox]] - pre-configured Savane virtual machine.
+* [[FrontEnd]] - Notes about the current frontend setup (git,cvs).
+* [[FrontEndDevelopmentSite]] - Setting up development sites for savannah.
+
+
+General information
+-------------------
+
+Savannah's web frontend (i.e. <https://savannah.gnu.org>)) is written in PHP.
+It uses a MySQL database running on `internal0` (for the new servers)
+or `internal` (for the existing servers).
+
+The code repository is called 'savane', available at
+<http://git.savannah.gnu.org/cgit/administration/savane.git> and
+<git://git.sv.gnu.org/administration/savane.git> (for developers with
+write-access: `git clone
+<USER>@git.sv.gnu.org:/srv/git/administration/savane.git`).
+
+At the time of this writing (September 2016), 'savane' and
+Savannah not 'installed' in a standard way (e.g. 'make install').
+Instead, files and settings are copied from old servers (e.g. the old
+VM of 'frontend.sv.gnu.org' to the new VM 'frontend0').  The MySQL
+database is not created from scratch, but is copied from existing
+servers (e.g. DB on internal to internal0).  The configuration files
+(e.g `/etc/savane` or the local copy in `/home/foobar/projects/savane-etc`)
+are copied from existing servers, and tweaked until they 'just work'.
+
+The PHP code runs directly from the cloned repository (i.e. if cloned
+into `$HOME/projects/savane`, Apache's php configuration should point to
+`$HOME/projects/savane/frontend/php/`).  The 'installation' above does
+not touch the PHP part.
+
+The current production website is <https://savannah.gnu.org>
+and is hosted on the 'old' VM (`frontend.sv.gnu.org`).
+
+The new production website is <https://frontend0.savannah.gnu.org>,
+and is hosted on the 'new' VM (`frontend0.sv.gnu.org`).
+The PHP code is in `frontend0:/usr/srv/savane`.
+The configuration is in `frontend0:/etc/savane`.
+
+
+
+Apache/PHP Configuration File Location
+--------------------------------------
+
+The environment variable `SAVANE_CONF` determines the location of the main
+PHP configuration file for savane. It could be set in Apache's configurations
+file (`frontend0:/etc/apache2/sites-available/sv.inc`) or in an `.htaccess`
+file (`/usr/src/savane/frontend/php/.htaccess`) with
+`SetEnv SAVANE_CONF /etc/savane`
+
+The `SAVANE_CONF` variable is used in `<savane>/frontend/php/include/init.php`
+to read a hard-coded file named `.savane.conf.php` from the specified 
directory.
+
+`init.php` also accepts a fall-back variable `SV_LOCAL_INC_PREFIX`, which
+is referenced here:
+`sites-available/sv.nongnu.inc:SetEnv SV_LOCAL_INC_PREFIX 
/etc/savane/nongnu-conf`.
+
+** On current production savannah, none of these are defined for the gnu side 
**
+
+The code in `init.php` falls back to:
+
+     # go back to default location
+     if (file_exists('/etc/savane/.savane.conf.php'))
+        include('/etc/savane/.savane.conf.php');
+
+
+Content files
+-------------
+
+Several sections of the displayed HTML content
+on the savannah website are set by external content files.
+
+See here for further details:
+<http://lists.gnu.org/archive/html/savannah-hackers-public/2016-09/msg00009.html>
+
+Savannah's licenses list are also stored in a content file,
+See this thread for more details:
+<http://lists.gnu.org/archive/html/savannah-hackers-public/2016-09/msg00001.html>
+
+
+Mail messages
+-------------
+
+Savannah's PHP code runs under apache as `www-data` user.
+Mail messages sent to `www-data` (cron and email errors)
+are now aliased to `root`
+(<http://lists.gnu.org/archive/html/savannah-hackers-public/2016-05/msg00058.html>).

Modified: trunk/sviki/SavannahHosts.mdwn
===================================================================
--- trunk/sviki/SavannahHosts.mdwn      2016-09-21 20:29:01 UTC (rev 225)
+++ trunk/sviki/SavannahHosts.mdwn      2016-09-21 20:29:08 UTC (rev 226)
@@ -548,11 +548,11 @@
     echo 'allow ^208\.118\.235\.77$' >> /etc/munin/munin-node.conf
     service munin-node restart
 
+See [[FrontEndSetup]], [[FrontEndDevelopmentSite]] for details
+about Apache/PHP/db configuration from the frontend's website.
+
 FIXME: Document max client limitations and configuration here.
 
-FIXME: Need to reference installing Savane here.  Perhaps referencing
-[[RunningSavaneLocally]]?
-
 FIXME: Need to transfer and test /etc/cron.d/sv_export cronjob.
 
 Host vcs0




reply via email to

[Prev in Thread] Current Thread [Next in Thread]