From 1e342dad3f8707fb27190513adc9efd75104e796 Mon Sep 17 00:00:00 2001 From: Chris Johns Date: Wed, 6 Mar 2019 22:22:57 +1100 Subject: user/test: Update with the latest test configuration format. --- user/testing/tests.rst | 110 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 96 insertions(+), 14 deletions(-) diff --git a/user/testing/tests.rst b/user/testing/tests.rst index 0b44d05..557cbc0 100644 --- a/user/testing/tests.rst +++ b/user/testing/tests.rst @@ -6,9 +6,9 @@ Test Banners ------------ All test output banners or strings are embedded in each test and the test -outputs them to the BSP's console as it executes. The RTEMS Tester captures the -BSP's console and uses this information to manage the state of the executing -test. The banner strings are: +outputs the banners to the BSP's console as it executes. The RTEMS Tester +captures the BSP's console and uses this information to manage the state of +the executing test. The banner strings are: .. _test-banner-begin: .. index:: test begin, TEST BEGIN @@ -65,10 +65,14 @@ test. The banner strings are: .. _test-states: .. index:: Test states -Test States ------------ +Test Controls +------------- + +The tests in the RTEMS kernel testsuite can be configured for each BSP. The +expected state of the test can be set as well as any configuration parameters +a test needs to run on a BSP. -The tests states are: +The test states are: .. index:: test state passed @@ -137,17 +141,46 @@ a tested is to ``pass``. If a test is known to fail it can have it's state set to ``expected-fail``. Setting tests that are known to fail to ``expected-fail`` lets everyone know a failure is not to be countered and consider a regression. -Expected test states are list in test configuration files that end with the -file extension ``.tcfg``. The testsuite supports global test configurations in -the ``testsuite/testdata`` directory. Global test states are applied to all -BSPs. BSPs can provide a test configuration that applies to just that BSP. +Expected test states are listed in test configuration files + +Test Configuration +^^^^^^^^^^^^^^^^^^ + +Tests can be configured for each BSP using test configuration files. These +files have the file extension ``.tcfg``. The testsuite supports global test +configurations in the ``testsuite/testdata`` directory. Global test states are +applied to all BSPs. BSPs can provide a test configuration that applies to +just that BSP and these files can include subsets of test configurations. + +The configuration supports: + +#. Including test configuration files to allow sharing of common + configurations. + +#. Excluding tests from being built that do not build for a BSP. + +#. Setting the test state if it is not ``passed``. -The test configuration file format is:: +#. Specifing a BSP specific build configuration for a test. - state: test test test +The test configuration file format is: -where ``test test test`` is a list of tests the state applies too. The ``state`` is one -of: +.. code-block:: none + + state: test + +where the ``state`` is state of the test and ``test`` is a comma separated +list of tests the state applies too. The configuration format is: + +.. code-block:: none + + flags: test: configuration + +where ``flags`` is the type of flags being set, the ``test`` is a comma +separate list of regular expresions that match the test the configuration +is applied too and the ``configuration`` is the string of flags. + +The ``state`` is one of: ``include`` The test list is the name of a test configuration file to include @@ -171,7 +204,56 @@ of: The tests listed are benchmarks. Benchmarks are flagged and not left to run to completion because they may take too long. +Specialized filtering using regular expressions is supported using: + +``rexclude`` + The test matching the regular expression are excluded. + +``rinclude`` + The tests matching the regular expression are included. + +By default all tests are included, specific excluded tests using the +``exclude`` state are excluded and cannot be included again. If a test +matching a ``rexclude`` regular it is excluded unless it is included using a +``rinclude`` regular expression. For example to build only the ``hello`` +sample application you can: + +.. code-block:: none + + rexclude .* + rinclude hello + +Test configuration flags can be applied to a range of tests using +flags. Currently only ``cflags`` is support. Tests need to support the +configuration for it to work. For example to configure a test: + +.. code-block:: none + + cflags: tm.*: -DTEST_CONFIG=42 + cflags: sp0[456]: -DA_SET_OF_TESTS=1 + +Flags setting are joined together and passed to the compiler's command +line. For example: + +.. code-block:: none + + cflags: tm.*: -DTEST_CONFIG=42 + cflags: tm03: -DTEST_TM03_CONFIG=1 + +would result in the command line to the test ``tm03`` being: + +.. code-block:: none + + -DTEST_CONFIG=42 -DTEST_TM03_CONFIG=1 + +Specific flags can be set for one test in a group. For example to set +a configuration for all timer tests and a special configuraiton for +one test: + +.. code-block:: none + cflags: (?!tm02)tm.*: -DTEST_CONFIG=one + cflags: tm02: -DTEST_CONFIG=two Test Builds ----------- -- cgit v1.2.3