Nightly regression testing

The nightly regression testing system consists of several shell scripts and some perl scripts. The goal is to check out a fresh tree, run "make check" on it, record the results and summarize the results to the team and to the web site.

Output can be found on adams, although the tests are actually run on another project machine.

How to setup the nightly build

The best way to do nightly testing is to setup a new account. We call the account "build" - you could call it something else, but there may still be some references to ~build in the scripts.

Files you need to know about

As few files as possible need to be extracted from the source tree - files are run from the source tree whenever possible. However, there are some bootstrap and configuration files that are necessary.

There are 7 files in testing/utils that are involved:

nightly-sample.sh

This is the root of the build process. This file should be copied out of the CVS tree, to $HOME/bin/nightly.sh of the build account. This file should be invoked from cron.

freeswan-regress-env-sample.sh

This file should be copied to $HOME/freeswan-regress-env.sh. It should be edited to localize the values. See below.

regress-cleanup.pl

This file needs to be copied to $HOME/bin/regress-cleanup.pl. It is invoked by the nightly file before doing anything else. It removes previous nights builds in order to free up disk space for the build about to be done.

teammail-sample.sh

A script used to send results email to the "team". This sample script could be copied to $HOME/bin/teammail.sh. This version will PGP encrypt all the output to the team members. If this script is used, then PGP will have to be properly setup to have the right keys.

regress-nightly.sh

This is the first stage of the nightly build. This stage will call other scripts as appropriate, and will extract the source code from CVS. This script should be copied to $HOME/bin/regress-nightly.sh

regress-stage2.sh

This is the second stage of the nightly build. It is called in place. It essentially sets up the UML setup in umlsetup.sh, and calls "make check".

regress-summarize-results.pl

This script will summarize the results from the tests to a permanent directory set by $REGRESSRESULTS. It is invoked from the stage2 nightly script.

regress-chart.sh

This script is called at the end of the build process, and will summarize each night's results (as saved into $REGRESSRESULTS by regress-summarize-results.pl) as a chart using gnuplot. Note that this requires at least gnuplot 3.7.2.

Configuring freeswan-regress-env.sh

For more info on KERNPOOL, UMLPATCH, BASICROOT and SHAREDIR, see User-Mode-Linux testing guide.

KERNPOOL

Extract copy of some kernel source to be used for UML builds

UMLPATCH

matching User-Mode-Linux patch.

BASICROOT

the root file system image (see User-Mode-Linux testing guide).

SHAREDIR=${BASICROOT}/usr/share

The /usr/share to use.

REGRESSTREE

A directory in which to store the nightly regression results. Directories will be created by date in this tree.

TCPDUMP=tcpdump-3.7.1

The path to the tcpdump to use. This must have crypto compiled in, and must be at least 3.7.1

KERNEL_RH7_2_SRC=/a3/kernel_sources/linux-2.4.9-13/

An extracted copy of the RedHat 7.2. kernel source. If set, then the packaging/rpm-rh72-install-01 test will be run, and an RPM will be built as a test.

KERNEL_RH7_3_SRC=/a3/kernel_sources/rh/linux-2.4.18-5

An extracted copy of the RedHat 7.3. kernel source. If set, then the packaging/rpm-rh73-install-01 test will be run, and an RPM will be built as a test.

NIGHTLY_WATCHERS="userid,userid,userid"

The list of people who should receive nightly output. This is used by teammail.sh

FAILLINES=128

How many lines of failed test output to include in the nightly output

PATH=$PATH:/sandel/bin export PATH

You can also override the path if necessary here.

CVSROOT=:pserver:anoncvs@ip212.xs4net.freeswan.org:/freeswan/MASTER

The CVSROOT to use. This example may work for anonymous CVS, but will be 12 hours behind the primary, and is still experimental

SNAPSHOTSIGDIR=$HOME/snapshot-sig

For the release tools, where to put the generated per-snapshot signature keys

LASTREL=1.97

the name of the last release branch (to find the right per-snapshot signature