Boost’s CMake-based build system provides regression testing via CTest, which comes as part of CMake. This regression testing system can be used by Boost developers to test their libraries locally and also by testers and users to submit regression tests to a CDash server, which collects and reports regression-testing results from different sites. This document assumes that the reader has already learned how to build and configure Boost using CMake.
The variable BUILD_TESTS is a comma-separated list of projects for which testing will be enabled, e.g.:
"accumulators;iostreams;variant"
or the string "ALL" for all projects, or the string "NONE" to disable testing.
Warning
if you pass this list from a unix shell, don’t forget to enclose the whole thing in quotes or escape the semicolons.
If you re-run the CMake configuration for Boost with BUILD_TESTS set to ALL, you will notice that configuration takes significantly longer when we are building all of the regression tests. This is due to the very large number of toplevel targets (thousands) that are created. Until boost’s testing scheme is reorganized to reduce this number, we anticipate that only testing nodes will want to test ALL, whereas developers will want to test the library under development and those that are dependent on it.
Warning
It is not recommended to set BUILD_TESTS to "ALL" when using Visual Studio generators. A very large number (thousands) of targets are generated and this can bring VS to grinding halt. To run all tests, choose the NMake generator, see Configuration for NMake.
Be sure to re-configure CMake once you are done tweaking these options, and generate makefiles or project files, as mentioned in Quickstart.
If you’re using a command-line configuration (nmake files, unix makefiles) you can simplify this process by passing the value of BUILD_TESTS on the command line, e.g.
cmake ../src -DBUILD_TESTS="mpi;graph_parallel"
Note
In Visual Studio, you should be prompted by the gui to reload the project. If you’re unlucky, you will be prompted a thousand times to reload each individual solution. For this reason, our current best recommendataion is to close and reopen the project if you rebuild Boost.sln.
Follow the same building process described in Quickstart. For Unix users, don’t forget the -i option to make (ignore errors), and also possibly -j 2 (or more) to run the build process in parallel. Building all of the regression tests for the Boost libraries can take a long time.
make -j2 -i
Note
If you change Boost source files in a way that affects your tests, you will need to rebuild to update the libraries and test executables before moving on to the next step.
Once regression tests have finished building,
at a command prompt, cd to the Boost binary directory. Then, run the command:
ctest
to execute all of the regression tests. The ctest executable comes with cmake. On Unix platforms, this is the same place where ccmake resides. On Windows platforms, it will be in C:\Program Files\CMake X.Y\bin. The ctest program should produce output like the following:
Start processing tests
Test project /Users/dgregor/Projects/boost-darwin
1/ 22 Testing any-any_test Passed
2/ 22 Testing any-any_to_ref_test Passed
3/ 22 Testing function-lib_function_test Passed
4/ 22 Testing function-function_n_test Passed
5/ 22 Testing function-allocator_test Passed
6/ 22 Testing function-stateless_test Passed
7/ 22 Testing function-lambda_test Passed
8/ 22 Testing function-function_test_fail1 ***Failed - supposed to fail
9/ 22 Testing function-function_test_fail2 ***Failed - supposed to fail
10/ 22 Testing function-function_30 Passed
11/ 22 Testing function-function_arith_cxx98 Passed
12/ 22 Testing function-function_arith_porta Passed
13/ 22 Testing function-sum_avg_cxx98 Passed
14/ 22 Testing function-sum_avg_portable Passed
15/ 22 Testing function-mem_fun_cxx98 Passed
16/ 22 Testing function-mem_fun_portable Passed
17/ 22 Testing function-std_bind_cxx98 Passed
18/ 22 Testing function-std_bind_portable Passed
19/ 22 Testing function-function_ref_cxx98 Passed
20/ 22 Testing function-function_ref_portabl Passed
21/ 22 Testing function-contains_test Passed
22/ 22 Testing function-contains2_test Passed
100% tests passed, 0 tests failed out of 22
Here, we have only enabled testing of the Boost.Any and Boost.Function libraries, by setting BUILD_TESTS to any;function.
Warning
Again, This ctest step runs the tests without first running a build. If you change a source file and run the ctest step you will see that no build is invoked.
To run just a subset of the tests, pass -R and a regular expression to ctest (see the output of ctest --help-full). For example, to run all of the Python tests, use:
ctest -R python
There is also a -E (exclude) option which does the inverse of -R. ctest --help shows the full list of options.
You will see a solution named RUN_TESTS. Build this to run the tests.
The testing subsystem adds toplevel targets to the build. On unix you can see them in the output of make help. For example some of the accumulators test targets look like this:
% make help | grep accum
... accumulators-tests-count
... accumulators-tests-covariance
... accumulators-tests-droppable
... accumulators-tests-error_of
... accumulators-tests-extended_p_square
... accumulators-tests-extended_p_square_quantile
Note that they are prefixed with the name of the project, a dash, and ‘tests’. Under visual studio you will see these targets in the ‘solution explorer’.
Donated by kitware, it is here:
Warning
This needs updating for git
The ctest command can be used by individual developers to test local changes to their libraries. The same program can also be used to build all of Boost, run its regression tests, and submit the results to a central server where others can view them. Currently, regression test results based on the CMake build system are displayed on the Dart server at http://www.cdash.org/CDashPublic/index.php?project=Boost.
To submit “experimental” results to the Dart server, configure a Boost binary tree by following the configuration instructions in the section Quickstart, and then enable regression testing via the BOOST_TESTS=ALL option, as described above. At this point, don’t build anything! We’ll let CTest do that work. You may want to customize some of the advanced CMake options, such as SITE (to give your site name), and MAKECOMMAND (which, for makefile targets, governs the top-level make call when building Boost). Finally, go into the Boost binary directory and execute:
ctest -D Experimental
CTest will then reconfigure Boost, build all of the Boost libraries and regression tests, execute the regression tests, and submit the results to the Dart dashboard at http://www.cdash.org/CDashPublic/index.php?project=Boost. Results submitted to the dashboard are usually browsable momentarily within a minute or two.
Here is a recommended setup.
Create a directory ctest on your test drone containing subdirectories for the branches you’ll be testing, in this case release and trunk.
boost/
ctest/
branches/
release/
continuous/
build/ <= run ctest here
src/ <= checkout to here
nightly/
build/ <= run ctest here
src/ <= checkout to here
and check out source to the directories listed above. We’ll do the release branch as an example:
% cd boost/ctest/branches/release
% svn co http://svn.boost.org/svn/boost/branches/release src
#
# lots of output
#
% mkdir continuous
% cd continuous
now configure your build, enabling testing. In this case I’ll also use an alternate compiler, from Intel:
% cmake ../src -DBUILD_TESTING=ON -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc
-- The C compiler identification is Intel
-- The CXX compiler identification is Intel
-- Check for working C compiler: /opt/intel/Compiler/11.0/083/bin/intel64/icc
-- Check for working C compiler: /opt/intel/Compiler/11.0/083/bin/intel64/icc -- works
(etc)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/troy/Projects/boost/ctest/release/continuous/build
Now run ctest -D Continuous in a loop:
% while true
while> do
while> ctest -D Continuous
while> sleep 600 # take it easy on the repository
while> done
Site: zinc
Build name: intel-11.0-linux
Create new tag: 20090514-2207 - Continuous
Start processing tests
Updating the repository
Updating the repository: /home/troy/Projects/boost/ctest/release/nightly/src
Use SVN repository type
Old revision of repository is: 53002
New revision of repository is: 53005
Gathering version information (one . per revision):
(etc)
If you add -V or -VV you’ll get a little more feedback about what is going on. On unix it is handy to do this via the utility screen.
Todo
Figure out how to do this on windows, encapsulate some of this scripting. Just use the ctest builtin scripting language.
Nightly testing should run triggered by a cron job or by Windows Task Scheduler or what-have-you. You will need,
but not a checkout of boost. CTest will do the checkout, build, test and submit on its own.
Create a directory to run in. As in the previous section, we’ll use boost/ctest/branches/release/nightly, which I’ll call $DIR. The CTest script should look like the following (you can copy/paste this into $DIR/CTestNightly.cmake
execute_process(COMMAND /bin/pwd
OUTPUT_VARIABLE PWD
OUTPUT_STRIP_TRAILING_WHITESPACE)
message(STATUS "Running nightly build in ${PWD}")
set(CTEST_SOURCE_DIRECTORY ${PWD}/src)
set(CTEST_BINARY_DIRECTORY ${PWD}/build)
# what cmake command to use for configuring this dashboard
set(CTEST_CMAKE_COMMAND "/usr/local/bin/cmake")
set(CTEST_CTEST_COMMAND "/usr/local/bin/ctest")
set(CTEST_CVS_COMMAND "svn")
set(CTEST_CVS_CHECKOUT "${CTEST_CVS_COMMAND} co https://svn.boost.org/svn/boost/branches/release ${CTEST_SOURCE_DIRECTORY}")
# which ctest command to use for running the dashboard
set(CTEST_COMMAND
"${CTEST_CTEST_COMMAND} -VV -D Experimental -A ${PWD}/notes.txt -O ctest.log"
)
####################################################################
# The values in this section are optional you can either
# have them or leave them commented out
####################################################################
# should ctest wipe the binary tree before running
set(CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
#
# this is the initial cache to use for the binary tree, be careful to escape
# any quotes inside of this string if you use it
#
# Yes you can pass cmake -DBUILD_WHATEVER=ON type options here.
#
set(CTEST_INITIAL_CACHE "
CMAKE_CXX_COMPILER:STRING=/opt/intel/Compiler/11.0/083/bin/intel64/icpc
CMAKE_C_COMPILER:STRING=/opt/intel/Compiler/11.0/083/bin/intel64/icc
SITE:STRING=zinc
MAKECOMMAND:STRING=make -i -j2
DART_TESTING_TIMEOUT:STRING=30
BUILD_TESTS:STRING=ALL
BUILD_EXAMPLES:STRING=ALL
CVSCOMMAND:FILEPATH=${CTEST_CVS_COMMAND}
")
You will need to customize several variables:
Now you’ll create a “notes” file, call it notes.txt. This will be visible from the dashboard. Add the output of, e.g:
gcc --version
uname -a
and the contents of the CTestNightly.cmake itself. Example:
**************** CMAKE DASHBOARD NOTES *****************
Notes file for CMake Nightly Dashboard.
This dashboard is being generated on an eight core, 64 bit
Ubuntu 9.04 machine with an intel c++ compiler.
Questions about this Dashboard should be directed to troy@resophonic.com
Linux zinc 2.6.27-11-generic #1 SMP Wed Apr 1 20:53:41 UTC 2009 x86_64 GNU/Linux
icpc (ICC) 11.0 20090318
Copyright (C) 1985-2009 Intel Corporation. All rights reserved.
--------------- Script file ---------------
(contents of CTestNightly.cmake)
Now run (as a cronjob or at the command line):
ctest -S CTestNightly.cmake
if you want extra verbosity add a -VV flag. You’ll see something like the following:
+ /opt/local/bin/ctest -VV -S CMakeDashBoard.cmake
* Extra verbosity turned on
Reading Script: /Users/troy/ctest/boost/release/nightly/CMakeDashBoard.cmake
Run cmake command: /opt/i3/ports/bin/cmake "/Users/troy/ctest/boost/release/nightly/src"
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
(etc)
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/troy/ctest/boost/release/nightly/build
Run ctest command: /opt/i3/ports/bin/ctest -VV -D Nightly -A /Users/troy/ctest/boost/release/nightly/notes.txt -O ctest.log
UpdateCTestConfiguration from :/Users/troy/ctest/boost/release/nightly/build/DartConfiguration.tcl
Parse Config file:/Users/troy/ctest/boost/release/nightly/build/DartConfiguration.tcl
Site: silver
Build name: gcc-4.0.1-macos
(etc, etc)
You’ll see it configure again, run... and sooner or later you’ll see your results on The Dashboard.