summaryrefslogtreecommitdiffstats
path: root/user
diff options
context:
space:
mode:
authorChris Johns <chrisj@rtems.org>2019-03-06 22:22:57 +1100
committerChris Johns <chrisj@rtems.org>2019-03-06 22:22:57 +1100
commit1e342dad3f8707fb27190513adc9efd75104e796 (patch)
tree81c7687e0db996db2b104f2465a7e375e641613f /user
parent1393ba0c97648fd2560a13c99aff3f1885cab64e (diff)
downloadrtems-docs-1e342dad3f8707fb27190513adc9efd75104e796.tar.bz2
user/test: Update with the latest test configuration format.
Diffstat (limited to 'user')
-rw-r--r--user/testing/tests.rst110
1 files changed, 96 insertions, 14 deletions
diff --git a/user/testing/tests.rst b/user/testing/tests.rst
index 0b44d05..557cbc0 100644
--- a/user/testing/tests.rst
+++ b/user/testing/tests.rst
@@ -6,9 +6,9 @@ Test Banners
------------
All test output banners or strings are embedded in each test and the test
-outputs them to the BSP's console as it executes. The RTEMS Tester captures the
-BSP's console and uses this information to manage the state of the executing
-test. The banner strings are:
+outputs the banners to the BSP's console as it executes. The RTEMS Tester
+captures the BSP's console and uses this information to manage the state of
+the executing test. The banner strings are:
.. _test-banner-begin:
.. index:: test begin, TEST BEGIN
@@ -65,10 +65,14 @@ test. The banner strings are:
.. _test-states:
.. index:: Test states
-Test States
------------
+Test Controls
+-------------
+
+The tests in the RTEMS kernel testsuite can be configured for each BSP. The
+expected state of the test can be set as well as any configuration parameters
+a test needs to run on a BSP.
-The tests states are:
+The test states are:
.. index:: test state passed
@@ -137,17 +141,46 @@ a tested is to ``pass``. If a test is known to fail it can have it's state set
to ``expected-fail``. Setting tests that are known to fail to ``expected-fail``
lets everyone know a failure is not to be countered and consider a regression.
-Expected test states are list in test configuration files that end with the
-file extension ``.tcfg``. The testsuite supports global test configurations in
-the ``testsuite/testdata`` directory. Global test states are applied to all
-BSPs. BSPs can provide a test configuration that applies to just that BSP.
+Expected test states are listed in test configuration files
+
+Test Configuration
+^^^^^^^^^^^^^^^^^^
+
+Tests can be configured for each BSP using test configuration files. These
+files have the file extension ``.tcfg``. The testsuite supports global test
+configurations in the ``testsuite/testdata`` directory. Global test states are
+applied to all BSPs. BSPs can provide a test configuration that applies to
+just that BSP and these files can include subsets of test configurations.
+
+The configuration supports:
+
+#. Including test configuration files to allow sharing of common
+ configurations.
+
+#. Excluding tests from being built that do not build for a BSP.
+
+#. Setting the test state if it is not ``passed``.
-The test configuration file format is::
+#. Specifing a BSP specific build configuration for a test.
- state: test test test
+The test configuration file format is:
-where ``test test test`` is a list of tests the state applies too. The ``state`` is one
-of:
+.. code-block:: none
+
+ state: test
+
+where the ``state`` is state of the test and ``test`` is a comma separated
+list of tests the state applies too. The configuration format is:
+
+.. code-block:: none
+
+ flags: test: configuration
+
+where ``flags`` is the type of flags being set, the ``test`` is a comma
+separate list of regular expresions that match the test the configuration
+is applied too and the ``configuration`` is the string of flags.
+
+The ``state`` is one of:
``include``
The test list is the name of a test configuration file to include
@@ -171,7 +204,56 @@ of:
The tests listed are benchmarks. Benchmarks are flagged and not left to
run to completion because they may take too long.
+Specialized filtering using regular expressions is supported using:
+
+``rexclude``
+ The test matching the regular expression are excluded.
+
+``rinclude``
+ The tests matching the regular expression are included.
+
+By default all tests are included, specific excluded tests using the
+``exclude`` state are excluded and cannot be included again. If a test
+matching a ``rexclude`` regular it is excluded unless it is included using a
+``rinclude`` regular expression. For example to build only the ``hello``
+sample application you can:
+
+.. code-block:: none
+
+ rexclude .*
+ rinclude hello
+
+Test configuration flags can be applied to a range of tests using
+flags. Currently only ``cflags`` is support. Tests need to support the
+configuration for it to work. For example to configure a test:
+
+.. code-block:: none
+
+ cflags: tm.*: -DTEST_CONFIG=42
+ cflags: sp0[456]: -DA_SET_OF_TESTS=1
+
+Flags setting are joined together and passed to the compiler's command
+line. For example:
+
+.. code-block:: none
+
+ cflags: tm.*: -DTEST_CONFIG=42
+ cflags: tm03: -DTEST_TM03_CONFIG=1
+
+would result in the command line to the test ``tm03`` being:
+
+.. code-block:: none
+
+ -DTEST_CONFIG=42 -DTEST_TM03_CONFIG=1
+
+Specific flags can be set for one test in a group. For example to set
+a configuration for all timer tests and a special configuraiton for
+one test:
+
+.. code-block:: none
+ cflags: (?!tm02)tm.*: -DTEST_CONFIG=one
+ cflags: tm02: -DTEST_CONFIG=two
Test Builds
-----------