Add docs builder scripts for the RTEMS servers.

9 files changed, 739 insertions, 0 deletions

diff --git a/docs/builder/README b/docs/builder/README
new file mode 100644
index 0000000..7e63cfd
--- /dev/null
+++ b/docs/builder/README
@@ -0,0 +1,187 @@
+RTEMS Documentation Web Site
+============================
+
+Chris Johns <chrisj@rtems.org>
+20th March 2017
+
+The following documents the RTEMS documentation website.
+
+Jail Access
+-----------
+
+The RTEMS Documentation website is a jail on services2.rtems.org.
+
+Login into services2.rtems.org:
+
+ $ ssh -A chrisj@services2.rtems.org
+
+Change to root to exec a shell in the jali:
+
+ [chrisj@service2 ~]$ jls
+    JID  IP Address      Hostname                      Path
+         1  140.211.10.142  lists.rtems.org            /usr/local/jail/lists.rtems.org
+      2  140.211.10.143  git.rtems.org                 /usr/local/jail/git.rtems.org
+      4  140.211.10.145  www.rtems.org                 /usr/local/jail/www.rtems.org
+      5  140.211.10.146  devel.rtems.org               /usr/local/jail/devel.rtems.org
+      6  140.211.10.150  dispatch.rtems.org            /usr/local/jail/dispatch.rtems.org
+      7  140.211.10.153  qualification.rtems.org       /usr/local/jail/qualification.rtems.org
+      8  140.211.10.144  docs.rtems.org                /usr/local/jail/docs.rtems.org
+ [chrisj@service2 ~]$ su
+ Password:
+ root@service2:/home/chrisj # jexec 8 /bin/sh
+ # su - chrisj
+ \u@docs#
+
+Locations
+---------
+
+The website configuration is:
+
+ /usr/local/etc/apache24/vhost/docs.rtems.org.conf
+
+The web pages are under:
+
+ /data/www/docs.rtems.org
+
+Website
+-------
+
+The website is held in the publically avaliable personal repo called
+`rtems-admin.git`. To update clone the repo:
+
+ # git clone git://git.rtems.org/chrisj/rtems-admin.git
+
+Build the website with the prefix being the location of the website in the
+jail:
+
+ # cd rtems-admin/docs.rtems.org/
+ # ./waf confgure --prefix=/data/www/docs.rtems.org
+ # ./waf
+
+To install the website do a waf install:
+
+ # ./waf install
+
+You can create a test website on a local machine with a suitable prefix. If you
+download and install the various release packages you will be able to loaded
+that documentation.
+
+The website is a mix of preset HTML and XML files called catalogues. A
+catalogue describes a documentation set and the website's Javascript that is
+loaded into user's browsers reads these confguration files from the server and
+dynamically creates the various web pages.
+
+There is specific HTML pieces that detail what is loaded on what pages. This
+hybrid mix lets us generate new documentation tarballso on a per commit basis
+and the website does need to be updated to reflect any local documentation repo
+changes.
+
+The HTML needs to be regenerated when a new release is made or a new release
+branch is created. This is not often and the over head to do is small.
+
+Legacy Documentation
+~~~~~~~~~~~~~~~~~~~~
+
+The lagacy documentation has been fitted into the new website by the website
+providing XML catalogue files for each release. These catalogues are generate
+when the website is built by waf. The configuration of these websites is held
+in configuration.ini. This dats is static and will not change so generating the
+website this way means a consistent user interface for what is old
+documentation.
+
+ReST Documentation
+~~~~~~~~~~~~~~~~~~
+
+The current documentation creates a catalogue and it is the catalogue being
+part of the documentation that gives us the needed flexibility. The
+documentation also contains an HTML coverpage which is based on the
+docs.rtems.org website so the look and feel is consistent.
+
+Configuration
+~~~~~~~~~~~~~
+
+The configuration.ini files is a number of sections. The following describes
+the sections.
+
+The `latest` section defines the latest release and the release key number be
+present in the `releases` section.
+
+The `branches` section describes the branches we are tracking.  The key is the
+name we used on the website and the value is the git label (I think). There
+must be a valid catalogue int the directory `branches/[branch]` where
+`[branch]` is the listed branch on the website. Scripts supporting the website
+manage the uploading of a branch update to the website.
+
+The `releases` section defines all releases where the key is the release label
+and the value is the releases directory. This is the directory in the release
+tarball. If you look at the releases you can see how this has evolved and an
+error or two.
+
+The `titles` section is collection of all titles for the manual parts in all
+the documentation. It is a common location for the naming so we get a
+consistent interface for the user.
+
+The remaining sections define each release. The 4.11 and later releases are
+nothing more than a `legacy` set to `no`. The documentation package's catalogue
+holds all the information.
+
+For legacy documentation you need to create a per release section which is the
+value field in the releases section. You can specify a template which as each
+release stream is the same. The PDF and HTML paths need to because they vary
+and with some errors there is no specific pattern. Finally the manual mix
+varies between major releases so a list of manuals that are present needs to be
+provided.
+
+Website Updating
+----------------
+
+The website is updated by documentation built outside of the docs.rtems.org
+jail. There is no sphinx or Tex installed.
+
+Releases are handled manually by updating the website repo, pulling in the
+changes, building it and then installing it as root. The documentation needs to
+be installed by hand up downloaded the release's tarball and then installing it
+to the releases directory.
+
+Note, 4.11.1 does not have a suitable catalogue so I created one.
+
+Updating branches currently requires 3 machines, a machine to build the
+documentation, the RTEMS FTP server to upload the tarball too and the
+docs.rtems.org machine to download the built tarball and to install the files.
+
+To complicate things more uploading to the FTP server is via SSH and currently
+I am using a key with a passphase. I am considering a key without a passphase
+which is limited to running rsync.
+
+The scripts used are held with this README and are broken down in pieces to
+allow easy testing and also flexibility to handle different path ways into
+machines.
+
+Building and Uploading
+~~~~~~~~~~~~~~~~~~~~~~
+
+The `rtems-docs-upload` is a top level script that loops in a shell with the
+SSH key added to an SSH agent. If a build has built any documentation it is
+uploaded to the RTEMS FTP server. The repo is checked every 10 minutes.
+
+The `rtems-docs-build-branches` builds each branch that has changed and
+packages the changes into a tarball. It also generates a tag file containing
+the branch and hash for each branch in the package.
+
+Downloading and Publishing
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `rtems-docs-cron` is a top level script that runs every 5 minutes. It
+downloads then publishes any changes.
+
+The `rtems-docs-download` script fetches the tags file from the FTP server and
+checks each tag again the tags currently published. If there is a difference in
+the file package is downloaeded and the branches unpacked.
+
+The script `rtems-docs-unpack-branches` unpacks the new documentation and
+creates a new branches website by selecting the new documentation if the tag's
+hash does not match or the current documentation. A new updated tags file is
+created.
+
+The `rtems-docs-publish` checks for a new branches website and if present it
+removes the current one and copies in the new one.
diff --git a/docs/builder/rtems-docs-build-branches b/docs/builder/rtems-docs-build-branches
new file mode 100755
index 0000000..7424238
--- /dev/null
+++ b/docs/builder/rtems-docs-build-branches
@@ -0,0 +1,155 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# Executables
+#
+git=/usr/local/bin/git
+
+#
+# Top directory
+#
+top=$PWD
+
+#
+# Directory of builds.
+#
+output=${top}/output
+
+#
+# The build tags files.
+#
+tags=${output}/.tags
+
+#
+# Branches to build and install as current
+#
+branches="master 4.11"
+
+#
+# Branch hash cache
+#
+hash_cache=${top}/git-hash-cache
+
+#
+# Git repository.
+#
+repo=rtems-docs.git
+
+#
+# Update path to include the user installed tools
+#
+PATH=$PATH:/usr/local/bin
+
+#
+# Logging.
+#
+log() {
+ echo $* >> ${log_file}
+}
+log_file=${top}/build.log
+for f in 9 8 7 6 5 4 3 2 1
+do
+ t=`expr $f + 1`
+ rm -f ${top}/build-${t}.log
+ if [ -f ${top}/build-${f}.log ]; then
+  mv ${top}/build-${f}.log ${top}/build-${t}.log
+ fi
+done
+if [ -f ${top}/build.log ]; then
+ mv ${top}/build.log ${top}/build-1.log
+fi
+rm -rf ${log_file}
+
+# Create or clone the repository and then check if a commit has been added on
+# each branch we are interesed in and if there is a commit build the doco and
+# install.
+
+log "RTEMS Documentation Build: " `date`
+
+log rm -f rtems-docs-latest.tar.xz rtems-docs-latest-tags
+rm -f rtems-docs-latest.tar.xz rtems-docs-latest-tags
+
+if [ ! -d ${hash_cache} ]; then
+ mkdir ${hash_cache}
+fi
+
+#
+# If there is no repo create it.
+#
+if [ ! -d ${repo} ]; then
+ log git clone git://git.rtems.org/${repo} ${repo}
+ ${git} clone git://git.rtems.org/${repo} ${repo} >> ${log_file} 2>&1
+fi
+
+#
+# If output exists we have an existing build so clean it first.
+#
+builds=0
+log rm -rf ${output}
+rm -rf ${output}
+
+cd ${repo}
+ #
+ # Fetch any updates and reset the repo to be safe.
+ #
+ log git fetch
+ ${git} fetch >> ${log_file} 2>&1
+ log git reset --hard
+ ${git} reset --hard >> ${log_file} 2>&1
+ for b in ${branches}
+ do
+  log git co ${b}
+  ${git} co ${b} >> ${log_file} 2>&1
+  log git pull
+  ${git} pull >> ${log_file} 2>&1
+  hash=`${git} log -n 1 --pretty=format:"%H"`
+  build=yes
+  if [ -f ${hash_cache}/${b} ]; then
+   if [ ${hash} = `cat ${hash_cache}/${b}` ]; then
+    build=no
+   fi
+  fi
+  log "Building ${b} : ${build}"
+  if [ ${build} = yes ]; then
+   prefix=${output}/${b}
+   log rm -rf ${prefix}
+   rm -rf ${prefix}
+   log mkdir -p ${prefix}
+   mkdir -p ${prefix}
+   log ./waf distclean
+   ./waf distclean >> ${log_file} 2>&1
+   log ./waf configure --pdf --singlehtml --prefix=${prefix}
+   ./waf configure --pdf --singlehtml --prefix=${prefix} >> ${log_file} 2>&1
+   log ./waf build install
+   ./waf build install >> ${log_file} 2>&1
+   echo ${hash} > ${hash_cache}/${b}
+   echo "${b} ${hash}" >> ${tags}
+   builds=$(expr ${builds} + 1)
+  fi
+ done
+ cd ..
+
+if [ ${builds} -gt 0 ]; then
+ cd output
+  #
+  # Package the builds.
+  #
+  tar Jcf ../rtems-docs-latest.tar.xz .tags *
+  cp .tags ../rtems-docs-latest-tags
+  cd ..
+fi
+
+exit 0
diff --git a/docs/builder/rtems-docs-build-cron b/docs/builder/rtems-docs-build-cron
new file mode 100755
index 0000000..a8e95ce
--- /dev/null
+++ b/docs/builder/rtems-docs-build-cron
@@ -0,0 +1,24 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# We run where the script is located.
+#
+cd $(dirname $0)
+
+. ./docs_builder/bin/activate
+./rtems-docs-upload
+
+exit 0
diff --git a/docs/builder/rtems-docs-cron b/docs/builder/rtems-docs-cron
new file mode 100755
index 0000000..8e49dde
--- /dev/null
+++ b/docs/builder/rtems-docs-cron
@@ -0,0 +1,24 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# We run where the script is located.
+#
+cd $(dirname $0)
+
+./rtems-docs-download
+./rtems-docs-publish
+
+exit 0
diff --git a/docs/builder/rtems-docs-download b/docs/builder/rtems-docs-download
new file mode 100755
index 0000000..e2b78a2
--- /dev/null
+++ b/docs/builder/rtems-docs-download
@@ -0,0 +1,92 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# Executables
+#
+cp=/bin/cp
+diff=/usr/bin/diff
+grep=/usr/bin/grep
+mkdir=/bin/mkdir
+rm=/bin/rm
+sed=/usr/bin/sed
+
+#
+# We run where the script is located.
+#
+cd $(dirname $0)
+
+#
+# Top directory
+#
+top=$PWD
+
+#
+# The current directory.
+#
+wwwdocs=/data/www/docs.rtems.org/branches
+
+#
+# Upload URL.
+#
+docsurl=/data/ftp/pub/rtems/people/chrisj/docs.rtems.org/latest
+
+#
+# Current tags.
+#
+current_tags=${wwwdocs}/.tags
+
+#
+# If no update directory create it.
+#
+if [ ! -d latest ]; then
+ ${rm} -rf latest
+ ${mkdir} latest
+fi
+
+cd latest
+ update=no
+ #
+ # Download the latest tags and compare the branches in the package with the
+ # current tags.
+ #
+ ${cp} ${docsurl}/rtems-docs-latest-tags rtems-docs-latest-tags
+ if [ -f ${current_tags} ]; then
+  exec 3<& 0
+  exec 0< rtems-docs-latest-tags
+  while read line
+  do
+   branch=$(echo ${line} | ${sed} -e 's/[[:space:]].*//g')
+   latest_hash=$(echo ${line} | ${sed} -e 's/.*[[:space:]]//g')
+   current_hash=$(cat ${current_tags} | ${grep} ${branch} | ${sed} -e 's/.*[[:space:]]//g')
+   if [ "${latest_hash}" != "${current_hash}" ]; then
+    update=yes
+   fi
+  done
+ else
+  update=yes
+ fi
+ if [ ${update} = yes ]; then
+  ${rm} -rf current
+  ${cp} -r /data/www/docs.rtems.org/branches current
+  ${cp} ${docsurl}/rtems-docs-latest.tar.xz rtems-docs-latest.tar.xz
+  set +e
+   ../rtems-docs-unpack-branches
+  set -e
+ fi
+ ${rm} -f rtems-docs-latest-tags
+ cd ..
+
+exit 0
diff --git a/docs/builder/rtems-docs-publish b/docs/builder/rtems-docs-publish
new file mode 100755
index 0000000..d8a5f30
--- /dev/null
+++ b/docs/builder/rtems-docs-publish
@@ -0,0 +1,41 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# Executables
+#
+cp=/bin/cp
+rm=/bin/rm
+
+#
+# The current directory.
+#
+current=/data/www/docs.rtems.org/branches
+
+#
+# The new docs
+#
+new=/usr/home/chrisj/docs/latest/new
+
+#
+# If there is a new directory we have something to publsh.
+#
+if [ -d ${new} ]; then
+ ${rm} -rf ${current}.tags ${current}/*
+ ${cp} -r ${new}/.tags ${new}/* ${current}/.
+ ${rm} -rf ${new}
+fi
+
+exit 0
diff --git a/docs/builder/rtems-docs-publish-cron b/docs/builder/rtems-docs-publish-cron
new file mode 100755
index 0000000..8e49dde
--- /dev/null
+++ b/docs/builder/rtems-docs-publish-cron
@@ -0,0 +1,24 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# We run where the script is located.
+#
+cd $(dirname $0)
+
+./rtems-docs-download
+./rtems-docs-publish
+
+exit 0
diff --git a/docs/builder/rtems-docs-unpack-branches b/docs/builder/rtems-docs-unpack-branches
new file mode 100755
index 0000000..0cdaf28
--- /dev/null
+++ b/docs/builder/rtems-docs-unpack-branches
@@ -0,0 +1,151 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# Executables
+#
+cp=/bin/cp
+curl=/usr/local/bin/curl
+diff=/usr/bin/diff
+mkdir=/bin/mkdir
+mv=/bin/mv
+tar=/usr/bin/tar
+touch=/usr/bin/touch
+rm=/bin/rm
+
+#
+# Top directory
+#
+top=$PWD
+
+#
+# Directory of the latest build.
+#
+latest=latest
+
+#
+# Directory of the current website build.
+#
+current=current
+
+#
+# Directory of the current website build.
+#
+new=new
+
+#
+# The build tags files.
+#
+latest_tags=${latest}/.tags
+current_tags=${current}/.tags
+new_tags=${new}/.tags
+
+#
+# Branches to build and install as current
+#
+branches="master 4.11"
+
+#
+# Logging.
+#
+log() {
+ echo $* >> ${log_file}
+}
+log_file=${top}/unpack.log
+for f in 4 3 2 1
+do
+ t=`expr $f + 1`
+ ${rm} -f ${top}/unpack-${t}.log
+ if [ -f ${top}/unpack-${f}.log ]; then
+  ${mv} ${top}/unpack-${f}.log ${top}/unpack-${t}.log
+ fi
+done
+if [ -f ${top}/unpack.log ]; then
+ ${mv} ${top}/unpack.log ${top}/unpack-1.log
+fi
+${rm} -rf ${log_file}
+
+# Unpack the latest build and check if the hashes match and if different update
+# the current with the latest.
+
+log "RTEMS Documentation Unpack: " `date`
+
+if [ ! -d ${current} ]; then
+ log "error: no current website found"
+ exit 1
+fi
+
+#
+# Remove any new website if there is one hanging about.
+#
+log ${rm} -rf ${new}
+${rm} -rf ${new}
+log ${mkdir} ${new}
+log ${cp} -r ${current} ${new}
+${cp} -r ${current} ${new}
+
+#
+# Unpack the latest website.
+#
+log ${rm} -rf ${latest}
+${rm} -rf ${latest}
+log ${mkdir} ${latest}
+${mkdir} ${latest}
+cd latest
+ log ${tar} Jxf ../rtems-docs-latest.tar.xz
+ ${tar} Jxf ../rtems-docs-latest.tar.xz
+ cd ..
+
+#
+# if no current tag file create an empty one.
+#
+if [ ! -f ${current_tags} ]; then
+ log touch ${current_tags}
+ ${touch} ${current_tags}
+fi
+
+if [ -f ${latest_tags} ]; then
+ #
+ # Check each branch to see if the tags match.
+ #
+ for b in ${branches}
+ do
+  latest_hash=$(cat ${latest_tags} | grep ${b} | sed -e 's/.* //')
+  current_hash=$(cat ${current_tags} | grep ${b} | sed -e 's/.* //')
+  log "Latest ${b} hash: ${latest_hash}"
+  log "Current ${b} hash: ${current_hash}"
+  if [ ! -z "${latest_hash}" \
+	 -a "${latest_hash}" != "${current_hash}" ]; then
+   src=${latest}/${b}
+   hash=${latest_hash}
+   if [ -d ${src} ]; then
+    log "New: ${b} from ${src} (${hash})"
+    log rm -rf ${new}/${b}
+    rm -rf ${new}/${b}
+    log cp -r ${src} ${new}/${b}
+    ${cp} -r ${src} ${new}/${b}
+    log "Update ${b} in tags"
+    cat ${new_tags} | grep -v ${b} > ${new_tags}.tmp
+    echo "${b} ${hash}" >> ${new_tags}.tmp
+    rm -f ${new_tags}
+    mv ${new_tags}.tmp ${new_tags}
+   fi
+  fi
+ done
+else
+ log "No tags found in the latest docs"
+fi
+
+exit 0
diff --git a/docs/builder/rtems-docs-upload b/docs/builder/rtems-docs-upload
new file mode 100755
index 0000000..7c217b3
--- /dev/null
+++ b/docs/builder/rtems-docs-upload
@@ -0,0 +1,41 @@
+#! /bin/sh
+#
+# RTEMS Documentation Project
+#
+# Copyright 2017 Chris Johns <chrisj@rtems.org>
+# All rights reserved
+#
+# Public domain
+#
+
+#
+# Exit on error.
+#
+set -e
+
+#
+# Executables
+#
+mkdir=/bin/mkdir
+
+#
+# We run where the script is located.
+#
+cd $(dirname $0)
+
+#
+# If no update directory create it.
+#
+if [ ! -d latest ]; then
+ ${mkdir} latest
+fi
+
+cd latest
+ ../rtems-docs-build-branches
+ if [ -f rtems-docs-latest.tar.xz ]; then
+  mv rtems-docs-latest.tar.xz rtems-docs-latest-tags \
+     /data/ftp/pub/rtems/people/chrisj/docs.rtems.org/latest/
+ fi
+ cd ..
+
+exit 0