[Savannah-cvs] [255] HowToAdminThiswiki + others: describe new sviki configuration

[assafgordon]

Revision: 255
          
http://svn.sv.gnu.org/viewvc/?view=rev&root=administration&revision=255
Author:   agn
Date:     2016-10-04 03:15:07 +0000 (Tue, 04 Oct 2016)
Log Message:
-----------
HowToAdminThiswiki + others: describe new sviki configuration

Modified Paths:
--------------
    trunk/sviki/HowToAdminThisWiki.mdwn
    trunk/sviki/MigrationChecklist.mdwn
    trunk/sviki/SavannahHosts.mdwn

Modified: trunk/sviki/HowToAdminThisWiki.mdwn
===================================================================
--- trunk/sviki/HowToAdminThisWiki.mdwn 2016-10-04 02:14:03 UTC (rev 254)
+++ trunk/sviki/HowToAdminThisWiki.mdwn 2016-10-04 03:15:07 UTC (rev 255)
@@ -1,100 +1,258 @@
 Wiki Administration
 ===================
-This wiki is based on ikiwiki.  Access is through Subversion (no
-web-based editing).  Specifically, to update the pages:
 
-0. Check out the subversion repository in the administration project.
-See <https://savannah.gnu.org/svn/?group=administration>.
+Savannah's WIKI is available at <https://savannah.gnu.org/maintenance>.
+Despite the name, it is more a static-website generator (like Jekyll)
+than a full-fledged wiki.
 
-0. Make edits as you please on those local files, which use MarkDown
-<http://ikiwiki.info/ikiwiki/formatting>.  (ikiwiki compiles the
-markdown into HTML, in `/var/www/sviki`, but we need/should never touch
-those files.)
+It is based on ikiwiki (hence the name). Updating is done solely
+throught Subversion, web-based editing is disabled. Savannah Administrators
+have write-access to the svn repository.
 
-0. Run `svn commit -m'log message'`.
+See [[RecentChanges]] for latest activity.
 
+See <http://svn.savannah.gnu.org/viewvc/trunk/sviki/?root=administration> for
+the actual commits.
+
+All commits are publically available at the 'savannah-cvs' mailing list
+<https://lists.gnu.org/mailman/listinfo/savannah-cvs>.
+
+
+Format and style
+----------------
+
+The pages are written in IkiWiki markdown format,
+for details see <https://ikiwiki.info/ikiwiki/formatting/> .
+
+The files use the `.mdwn` extension.
+
+Diagrams are welcomed, and should be included as PNG files.
+Please be sure to include an editable source for the diagram as
+well (e.g. GNU Dia files). Do not use non-free file formats.
+
+
+
+One-off contributions
+---------------------
+
+We welcome improvements to the wiki!
+
+Use `svn`  to anonymously checkout the repository:
+
+    svn co svn://svn.savannah.gnu.org/administration/trunk/sviki
+
+Edit your changes, and create a diff file with:
+
+    svn diff > newstuff.patch
+
+Then email your suggestion to address@hidden .
+
+It is also possible to access Subversion repositories using
+Git through `git-svn`, which is an optional addition to the git suite of tools.
+
+Anonymously clone the repository:
+
+    git svn clone --stdlayout svn://svn.savannah.gnu.org/administration/
+
+Make your changes and create a diff using:
+
+    git diff > newstuff.patch
+
+Then email your suggestion to address@hidden .
+
+
+
+
+Write access (SVN)
+------------------
+
+Savannah administrators with write access to the savannah repositories
+can checkout the wiki pages using:
+
+    svn co svn+ssh://<USERNAME>@svn.savannah.gnu.org/administration/
+
+To fetch the latest changes to an existing repository, run:
+
+    svn update
+
+Make your changes, then commit and push with:
+
+    svn commit -m'log message'
+
 The result should show up under <http://savannah.gnu.org/maintenance/>
 in a few seconds, if all goes well.  Check [[RecentChanges]] if you
 like.
 
 
-Using Git
----------
-The repository is a Subversion repository.  It is possible to access
-Subversion repositories using Git through `git-svn`, which
-is an optional addition to the git suite of tools.
 
-0. git svn clone --stdlayout svn+ssh://svn.savannah.gnu.org/administration
+Write access (GIT)
+------------------
 
-0. Edit the local sandbox of files normally.
+It is possible to access Subversion repositories using
+Git through `git-svn`, which is an optional addition to the git suite of tools.
 
-0. Run `git commit` to commit local changes to the local git repository.
+Clone the repository:
 
-0. Run `git svn rebase` to fetch and rebase your local changes on top
-of any new upstream versions.  (Use `git svn fetch` to only fetch
-upstream changes but not rebase merge them.)
+    git svn clone --stdlayout 
svn+ssh://<USERNAME>@svn.savannah.gnu.org/administration
 
-0. Run `git svn dcommit` to publish locally committed changes to the
-public Subversion repository.  If due to a race with another upstream
-svn committer there is a conflict then `git svn rebase` again to pull
-their new upstream changes.  Then push again with `git svn dcommit`.
-(This is the same as the `svn update` and `svn commit`.)
+To get the latest changes (of an existing cloned repository) and
+rebase your local repository, run:
 
+    git svn rebase
 
-Wiki configuration
-------------------
-In the file `frontend:~sviki/ikiwiki.setup`.  At present, we are using
-the default (quite minimal) configuration, with no extra plugins.
+Make your changes, the commit with:
 
-That file is in RCS, for simplest version control.  Anyone in the
-`staff` group can edit it, which should be anyone with shell access to
-the machine.  (Savannah administrators are welcome to have shell
-logins, but Savannah users do not, for obvious security reasons.)
+    git commit -a -m "log message"
 
-The web server configuration bits are in
-`frontend:/etc/apache2/sites-available/sv.inc`.
+And push with:
 
+    git svn dcommit
 
-How it works
-------------
-Our ikiwiki setup on Savannah is unusual.  ikiwiki expects the source
-repository and wiki to live on the same machine, and thus permissions
-are simply a matter of having the files be writable to the desired Unix
-users.
 
-For Savannah, we very much wanted the source repository to be accessible
-in the standard Savannah ways, that is, through the `administration`
-project (<http://savannah.gnu.org/projects/administration>, which holds
-all the live Savannah stuff which is checked into version control).
-That way, access would be controlled just as to any other source
-repository.  We used the Subversion repository in `administration` as
-the wiki source partly because it was otherwise unused and partly
-because Karl (who did all this) likes Subversion.
+Use `git svn fetch` to only fetch upstream changes but not rebase merge them.
 
-Well, using a Savannah project means that source repository lives on one
-host (vcs) while the web-accessible wiki lives on another (frontend).
-And frontend is where the actual refresh is run, using
-`~sviki/sviki-refresh`.  This necessitated the following:
+If due to a race with another upstream svn committer there is a
+conflict then `git svn rebase` again to pull their new upstream
+changes.  Then push again with `git svn dcommit`.  (This is the same
+as the `svn update` and `svn commit`.)
 
-0. A post-commit hook on vcs which ssh's back to frontend to do the
-refresh.  This is in
-`vcs:/srv/svn/administration/hooks/post-commit.d/sv_ikiwiki_refresh`.
 
-0. A pseudo-user `sviki` on vcs, frontend, and mgt, with its own ssh key
-to allow communication (in `/etc/ssh/authorized_keys/sviki`).
+Wiki Generation - NEW VMs
+-------------------------
 
-0. An empty directory `vcs:/srv/svn/.ssh` because ... it's a long story.
-ssh unconditionally tries to create the user's `~/.ssh` directory, even
-when (as here) it will not be used for anything.  On vcs, the
-post-commit is running as the user who did the commit, which is to say,
-a normal Savannah user who does not have a real home directory but
-exists only in the mysql database.  The users' homes are arbitrarily set
-to `/srv/svn` instead, so that is where the `.ssh` needs to exist.
-Thankfully ssh does not initially require specific permissions or
-ownership on it; as long as it exists, ssh won't try to create it, and
-then it will never be accessed again (due to the options with which
-`sv_ikiwiki_refresh` invokes ssh).
+### On VCS
 
+The repository is in `vcs:/srv/svn/administration`. A post-commit hook
+`vcs:/srv/svn/administration/hooks/post-commit.d/sv-ikiwiki-refresh-2`
+will be executed after a new SVN commit. The script simply triggers
+an update on `frontend0`:
+
+    $ cat sv-ikiwiki-refresh-2
+    #!/bin/sh
+    curl https://frontend0.savannah.gnu.org/update-sviki/update.php
+
+### On Frontend0
+
+The apache configuration file
+`frontend0:/etc/apache2/sites-available/ikiwiki.inc` contains the following:
+
+    # Directory with update trigger script.
+    Alias /update-sviki /var/www/sviki/php/
+
+    # Directory with static HTML files, auto-generated from wiki pages.
+    Alias /maintenance /var/www/sviki/www
+
+The update PHP script `frontend0:/var/www/sviki/php/update.php` writes
+a message to the logs, and runs `frontend0:/var/www/sviki/sviki-refresh`.
+
+The update script `sviki-refresh` updates the wiki with the following
+commands (the actual script is slightly longer, but conceptually the same):
+
+    cd /var/www/sviki/src/sviki
+    svn -q update
+    ikiwiki --setup /var/www/sviki/ikiwiki.setup --refresh
+
+The checked-out SVN repository is in `frontend0:/var/www/sviki/src`.
+The generated HTML files are in `frontend0:/var/www/sviki/www`.
+
+
+### Troubleshooting
+
+0.  To force a rebuild, delete the checked-out SVN repository
+    on `frontend0` and re-run the update script as user `www-data`:
+
+        cd /var/www/sviki
+        rm -r src/sviki
+        sudo -u www-data ./sviki-refresh
+
+    This will run `svn co` again, and will run `ikiwiki --rebuild --verbose`
+    to rebuild all files in `/var/www/sviki/www`.
+
+0.  To trigger an update, run:
+
+        curl https://frontend0.savannah.gnu.org/update-sviki/update.php
+
+    There should be no output. To check for possible apache misconfiguration,
+    check the server's HTTP output headers:
+
+        curl -D /dev/stderr 
https://frontend0.savannah.gnu.org/update-sviki/update.php
+
+    Apache's configuration allows updating only from
+    `frontend`,`frontend0`,`vcs0`,`vsc`,`mgt`,`mgt0`. Other hosts
+    should get 'HTTP 403 forbidden'.
+
+0.  The `update.php` script writes log messages to apache's error/php log.
+    Look for them in `frontend0:/var/log/apache/error.log`:
+
+        [Mon Oct 03 22:14:03.443969 2016] [:error] [pid 4562] [client 
208.118.235.72:46847] sviki udpate (start)
+        [Mon Oct 03 22:14:03.755440 2016] [:error] [pid 4562] [client 
208.118.235.72:46847] sviki update (end) rc =
+
+
+
+Wiki Initial Setup - New VMs
+----------------------------
+
+On VCS/VCS0: no setup is necessary.
+
+On FrontEnd0:
+
+Install IkiWiki + required packages:
+
+    apt-get install -y subversion ikiwiki
+    cpan -i XML/SAX
+    cpan -i XML::Simple
+
+Create the sviki directories (as root):
+
+    cd /var/www
+    mkdir sviki ; cd sviki
+    mkdir src www php
+    mkdir .subversion    # this will be the configdir for svn
+
+The sviki updates run as `www-data`, setup permissions:
+
+    chgrp www-data src www .subversion
+    chmod g+ws src www .subversion
+
+Copied `ikiwiki.setup` from (the old) `frontend`, modified the following:
+
+    srcdir => '/var/www/sviki/src/sviki/',
+    destdir => '/var/www/sviki/www/',
+    rcs => '',
+
+Created `/var/ww/sviki/sviki-refresh`:
+
+    #!/bin/sh
+
+    SVNDIR=/var/www/sviki/.subversion/
+
+    cd /var/www/sviki/src
+    if ! test -d sviki ; then
+        # first time check out
+        svn --config-dir="$SVNDIR" co 
svn://svn.savannah.gnu.org/administration/trunk/sviki
+
+        # force a rebuild for a new checkout
+        method="--rebuild --verbose"
+    else
+        # just an update
+        ( cd sviki ; svn --config-dir="$SVNDIR" -q update )
+        method="--refresh"
+    fi
+    ikiwiki --setup /var/www/sviki/ikiwiki.setup $method
+
+
+The result is the following:
+
+    # ls -la
+    drwxrwsr-x   3 root www-data  4096 Oct  3 21:11 .subversion
+    -r--r--r--   1 root root     10047 Oct  3 21:24 ikiwiki.setup
+    drwxr-xr-x   2 root root      4096 Oct  3 22:10 php
+    drwxrwsr-x   3 root www-data  4096 Oct  3 21:26 src
+    -rwxr-xr-x   1 root root       866 Oct  3 22:13 sviki-refresh
+    drwxrwsr-x 161 root www-data 12288 Oct  3 21:26 www
+
+
+
 Running this wiki locally
 -------------------------
 To edit and review changes locally (before commiting, or submitting a patch
@@ -166,6 +324,66 @@
     $ ikiwiki --verbose ./administration/trunk/sviki /tmp/sviki
 
 
+
+
+Wiki configuration - current VMs
+--------------------------------
+
+In the file `frontend:~sviki/ikiwiki.setup`.  At present, we are using
+the default (quite minimal) configuration, with no extra plugins.
+
+That file is in RCS, for simplest version control.  Anyone in the
+`staff` group can edit it, which should be anyone with shell access to
+the machine.  (Savannah administrators are welcome to have shell
+logins, but Savannah users do not, for obvious security reasons.)
+
+The web server configuration bits are in
+`frontend:/etc/apache2/sites-available/sv.inc`.
+
+
+How it works - current VMs
+--------------------------
+
+Our ikiwiki setup on Savannah is unusual.  ikiwiki expects the source
+repository and wiki to live on the same machine, and thus permissions
+are simply a matter of having the files be writable to the desired Unix
+users.
+
+For Savannah, we very much wanted the source repository to be accessible
+in the standard Savannah ways, that is, through the `administration`
+project (<http://savannah.gnu.org/projects/administration>, which holds
+all the live Savannah stuff which is checked into version control).
+That way, access would be controlled just as to any other source
+repository.  We used the Subversion repository in `administration` as
+the wiki source partly because it was otherwise unused and partly
+because Karl (who did all this) likes Subversion.
+
+Well, using a Savannah project means that source repository lives on one
+host (vcs) while the web-accessible wiki lives on another (frontend).
+And frontend is where the actual refresh is run, using
+`~sviki/sviki-refresh`.  This necessitated the following:
+
+0. A post-commit hook on vcs which ssh's back to frontend to do the
+refresh.  This is in
+`vcs:/srv/svn/administration/hooks/post-commit.d/sv_ikiwiki_refresh`.
+
+0. A pseudo-user `sviki` on vcs, frontend, and mgt, with its own ssh key
+to allow communication (in `/etc/ssh/authorized_keys/sviki`).
+
+0. An empty directory `vcs:/srv/svn/.ssh` because ... it's a long story.
+ssh unconditionally tries to create the user's `~/.ssh` directory, even
+when (as here) it will not be used for anything.  On vcs, the
+post-commit is running as the user who did the commit, which is to say,
+a normal Savannah user who does not have a real home directory but
+exists only in the mysql database.  The users' homes are arbitrarily set
+to `/srv/svn` instead, so that is where the `.ssh` needs to exist.
+Thankfully ssh does not initially require specific permissions or
+ownership on it; as long as it exists, ssh won't try to create it, and
+then it will never be accessed again (due to the options with which
+`sv_ikiwiki_refresh` invokes ssh).
+
+
+
 Initial setup
 -------------
 For posterity, here is how the initial setup went.
@@ -280,12 +498,4 @@
     rc5.d/K02zope2.10 -> ../init.d/zope2.10
     rc6.d/K02zope2.10 -> ../init.d/zope2.10
 
-And we were back in business.
-
-
-Test Please Ignore
-------------------
-
-* Testing new Sviki installation and auto-updates.
-* Another test
-* yet another
+And we were back in business.
\ No newline at end of file

Modified: trunk/sviki/MigrationChecklist.mdwn
===================================================================
--- trunk/sviki/MigrationChecklist.mdwn 2016-10-04 02:14:03 UTC (rev 254)
+++ trunk/sviki/MigrationChecklist.mdwn 2016-10-04 03:15:07 UTC (rev 255)
@@ -39,9 +39,6 @@
 
 * Apache configuration: rename all mentions of `frontend0` to `frontned`.
 
-* Sviki: install according to [[HowToAdminThisWiki]] (See 'Initial Setup'
-  section).
-
 * SSL Certificates: Ensure shortcut domains (e.g. 'sv.gnu.org') work
   and redirect successfully with HTTPS, and provide unredirected `.well-known`
   directory for HTTP. All domains should be included
@@ -50,7 +47,9 @@
   See [[FrontEndHostRedirection]] for redirection details and apache example.
   See [[https]] for certbot/ssl informaiton.
 
+* Sviki - *DONE*. See [[HowToAdminThisWiki]].
 
+
 Internal0 (MySQL + DNS)
 -----------------------
 

Modified: trunk/sviki/SavannahHosts.mdwn
===================================================================
--- trunk/sviki/SavannahHosts.mdwn      2016-10-04 02:14:03 UTC (rev 254)
+++ trunk/sviki/SavannahHosts.mdwn      2016-10-04 03:15:07 UTC (rev 255)
@@ -570,6 +570,9 @@
 See [[FrontEndSetup]], [[FrontEndDevelopmentSite]] for details
 about Apache/PHP/db configuration from the frontend's website.
 
+See [[HowToAdminThisWiki]] for wiki-related configuration settings
+on frontend0.
+
 FIXME: Document max client limitations and configuration here.
 
 FIXME: Need to transfer and test /etc/cron.d/sv_export cronjob.