From 21fa28c7ad81d59752b7c546a5c753c887c2eae6 Mon Sep 17 00:00:00 2001 From: Sebastian Huber Date: Tue, 21 Jul 2020 16:38:49 +0200 Subject: eng: Update test framework chapter Document the dynamic text fixtures, utility functions, and the interrupt test support. Reorder some sections and reword some paragraphs based on review comments. Update #3199. --- eng/index.rst | 4 +- eng/test-framework.rst | 412 +++++++++++++++++++++++++++++++----------- images/eng/interrupt-test.odg | Bin 0 -> 14829 bytes images/eng/interrupt-test.pdf | Bin 0 -> 14153 bytes images/eng/interrupt-test.png | Bin 0 -> 75272 bytes 5 files changed, 313 insertions(+), 103 deletions(-) create mode 100644 images/eng/interrupt-test.odg create mode 100644 images/eng/interrupt-test.pdf create mode 100644 images/eng/interrupt-test.png diff --git a/eng/index.rst b/eng/index.rst index 8f91c5e..f6b02ec 100644 --- a/eng/index.rst +++ b/eng/index.rst @@ -11,8 +11,8 @@ RTEMS Software Engineering (|version|) .. topic:: Copyrights and License - | |copy| 2018, 2019 embedded brains GmbH - | |copy| 2018, 2019 Sebastian Huber + | |copy| 2018, 2020 embedded brains GmbH + | |copy| 2018, 2020 Sebastian Huber | |copy| 1988, 2015 On-Line Applications Research Corporation (OAR) .. include:: ../common/license.rst diff --git a/eng/test-framework.rst b/eng/test-framework.rst index b6411b5..1cdf018 100644 --- a/eng/test-framework.rst +++ b/eng/test-framework.rst @@ -1,7 +1,7 @@ .. SPDX-License-Identifier: CC-BY-SA-4.0 -.. Copyright (C) 2018, 2019 embedded brains GmbH -.. Copyright (C) 2018, 2019 Sebastian Huber +.. Copyright (C) 2018, 2020 embedded brains GmbH +.. Copyright (C) 2018, 2020 Sebastian Huber Software Test Framework *********************** @@ -16,6 +16,8 @@ features: * Implemented in standard C11 +* Tests can be written in C or C++ + * Runs on at least FreeBSD, MSYS2, Linux and RTEMS * Test runner and test case code can be in separate translation units @@ -67,11 +69,11 @@ A `test suite` is a collection of test cases. A `test case` consists of individual test actions and checks. A `test check` determines if the outcome of a test action meets its expectation. A `test action` is a program sequence with an observable outcome, for example a function invocation with a return -status. If the test action outcome is all right, then the test check passes, -otherwise the test check fails. The test check failures of a test case are -summed up. A test case passes, if the failure count of this test case is zero, -otherwise the test case fails. The test suite passes if all test cases pass, -otherwise it fails. +status. If a test action produces the expected outcome as determined by the +corresponding test check, then this test check passes, otherwise this test +check fails. The test check failures of a test case are summed up. A test +case passes, if the failure count of this test case is zero, otherwise the test +case fails. The test suite passes if all test cases pass, otherwise it fails. Test Cases ---------- @@ -89,7 +91,7 @@ body: The test case `name` must be a valid C designator. The test case names must be unique within the test suite. Just link modules with test cases to the test runner to form a test suite. The test cases are automatically registered via -static constructors. +static C constructors. .. code-block:: c :caption: Test Case Example @@ -144,13 +146,43 @@ macro followed by a function body: The test case `name` must be a valid C designator. The test case names must be unique within the test suite. The `fixture` must point to a statically -initialized read-only object of type `T_fixture`. The test fixture -provides methods to setup, stop and tear down a test case. A context is passed -to the methods. The initial context is defined by the read-only fixture -object. The context can be obtained by the `T_fixture_context()` -function. It can be set within the scope of one test case by the -`T_set_fixture_context()` function. This can be used for example to -dynamically allocate a test environment in the setup method. +initialized read-only object of type `T_fixture`. + +.. code-block:: c + + typedef struct T_fixture { + void (*setup)(void *context); + void (*stop)(void *context); + void (*teardown)(void *context); + void (*scope)(void *context, char *buffer, size_t size); + void *initial_context; + } T_fixture; + +The test fixture provides methods to setup, stop, and teardown a test case as +well as the scope for log messages. A context is passed to each of the +methods. The initial context is defined by the read-only fixture object. The +context can be obtained by the `T_fixture_context()` function. The context can +be changed within the scope of one test case by the `T_set_fixture_context()` +function. The next test case execution using the same fixture will start again +with the initial context defined by the read-only fixture object. Setting the +context can be used for example to dynamically allocate a test environment in +the setup method. + +The test case fixtures of a test case are organized as a stack. Fixtures can +be dynamically added to a test case and removed from a test case via the +`T_push_fixture()` and `T_pop_fixture()` functions. + +.. code-block:: c + + void *T_push_fixture(T_fixture_node *node, const T_fixture *fixture); + + void T_pop_fixture(void); + +The `T_push_fixture()` function needs an uninitialized fixture node which must +exist until `T_pop_fixture()` is called. It returns the initial context of the +fixture. At the end of a test case all pushed fixtures are popped +automatically. A call of `T_pop_fixture()` invokes the teardown method of the +fixture and must correspond to a previous call to `T_push_fixture()`. .. code-block:: c :caption: Test Fixture Example @@ -237,9 +269,9 @@ dynamically allocate a test environment in the setup method. Test Case Planning ------------------ -Each non-quiet test check fetches and increments the test step counter -atomically. For each test case execution the planned steps can be specified -with the `T_plan()` function. +A non-quiet test check fetches and increments the test step counter atomically. +For each test case execution the planned steps can be specified with the +`T_plan()` function. .. code-block:: c @@ -318,7 +350,7 @@ execution follows exactly the planned steps. Test Case Resource Accounting ----------------------------- -The framework can check if various resources are leaked during a test case +The framework can check if various resources have leaked during a test case execution. The resource checkers are specified by the test run configuration. On RTEMS, checks for the following resources are available @@ -452,13 +484,14 @@ test case execution. You can provide an optional destroy function to Test Case Destructors --------------------- -You can add test case destructors with `T_add_destructor()`. They are called -automatically at the test case end before the resource accounting takes place. -Optionally, a registered destructor can be removed before the test case end -with `T_remove_destructor()`. The `T_destructor` structure of a destructor -must exist after the return from the test case body. Do not use stack memory -or dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for -the `T_destructor` structure. +You can add test case destructors with `T_add_destructor()`. The destructors +are called automatically at the test case end before the resource accounting +takes place. Optionally, a registered destructor can be removed before the +test case end with `T_remove_destructor()`. The `T_destructor` structure of a +destructor must exist after the return from the test case body. It is +recommended to use statically allocated memory. Do not use stack memory or +dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for the +`T_destructor` structure. .. code-block:: c @@ -497,16 +530,31 @@ the `T_destructor` structure. Test Checks ----------- -A `test check` determines if the actual value presented to the test check meets -its expectation. The actual value should represent the outcome of a test -action. If the actual value is all right, then the test check passes, -otherwise the test check fails. A failed test check does not stop the test -case execution immediately unless the `T_assert_*()` test variant is used. -Each test check increments the test step counter unless the `T_quiet_*()` test -variant is used. The test step counter is initialized to zero before the test -case begins to execute. The `T_step_*(step, ...)` test check variants verify -that the test step counter is equal to the planned test step value, otherwise -the test check fails. +A `test check` determines if the actual value presented to the test check has +the expected properties. The actual value should represent the outcome of a +test action. If a test action produces the expected outcome as determined by +the corresponding test check, then this test check passes, otherwise this test +check fails. A failed test check does not stop the test case execution +immediately unless the `T_assert_*()` test variant is used. Each test check +increments the test step counter unless the `T_quiet_*()` test variant is used. +The test step counter is initialized to zero before the test case begins to +execute. The `T_step_*(step, ...)` test check variants verify that the test +step counter is equal to the planned test step value, otherwise the test check +fails. + +Test Check Variant Conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `T_quiet_*()` test check variants do not increment the test step counter +and only print a message if the test check fails. This is helpful in case a +test check appears in a tight loop. + +The `T_step_*(step, ...)` test check variants check in addition that the test +step counter is equal to the specified test step value, otherwise the test +check fails. + +The `T_assert_*()` and `T_step_assert_*(step, ...)` test check variants stop +the current test case execution if the test check fails. Test Check Parameter Conventions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -557,21 +605,10 @@ lt If the actual value satisfies the test check condition, then the test check passes, otherwise it fails. -Test Check Variant Conventions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The `T_quiet_*()` test check variants do not increment the test step counter -and only print a message if the test check fails. This is helpful in case a -test check appears in a tight loop. - -The `T_step_*(step, ...)` test check variants check in addition that the test -step counter is equal to the specified test step value, otherwise the test -check fails. - -The `T_assert_*()` and `T_step_assert_*(step, ...)` test check variants stop -the current test case execution if the test check fails. +Test Check Type Conventions +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The following names for test check type variants are used: +The following names for test check types are used: ptr The test value must be a pointer (`void *`). @@ -654,6 +691,57 @@ ssz sz The test value must be of type `size_t`. +Integers +~~~~~~~~ + +Let `xyz` be the type variant which shall be one of `schar`, `uchar`, `short`, +`ushort`, `int`, `uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`, +`u16`, `i32`, `u32`, `i64`, `u64`, `iptr`, `uptr`, `ssz`, and `sz`. + +Let `I` be the type name which shall be compatible to the type variant. + +The following test checks for integers are available: + +.. code-block:: c + + void T_eq_xyz(I a, I e); + void T_assert_eq_xyz(I a, I e); + void T_quiet_eq_xyz(I a, I e); + void T_step_eq_xyz(unsigned int step, I a, I e); + void T_step_assert_eq_xyz(unsigned int step, I a, I e); + + void T_ne_xyz(I a, I e); + void T_assert_ne_xyz(I a, I e); + void T_quiet_ne_xyz(I a, I e); + void T_step_ne_xyz(unsigned int step, I a, I e); + void T_step_assert_ne_xyz(unsigned int step, I a, I e); + + void T_ge_xyz(I a, I e); + void T_assert_ge_xyz(I a, I e); + void T_quiet_ge_xyz(I a, I e); + void T_step_ge_xyz(unsigned int step, I a, I e); + void T_step_assert_ge_xyz(unsigned int step, I a, I e); + + void T_gt_xyz(I a, I e); + void T_assert_gt_xyz(I a, I e); + void T_quiet_gt_xyz(I a, I e); + void T_step_gt_xyz(unsigned int step, I a, I e); + void T_step_assert_gt_xyz(unsigned int step, I a, I e); + + void T_le_xyz(I a, I e); + void T_assert_le_xyz(I a, I e); + void T_quiet_le_xyz(I a, I e); + void T_step_le_xyz(unsigned int step, I a, I e); + void T_step_assert_le_xyz(unsigned int step, I a, I e); + + void T_lt_xyz(I a, I e); + void T_assert_lt_xyz(I a, I e); + void T_quiet_lt_xyz(I a, I e); + void T_step_lt_xyz(unsigned int step, I a, I e); + void T_step_assert_lt_xyz(unsigned int step, I a, I e); + +An automatically generated message is printed in case the test check fails. + Boolean Expressions ~~~~~~~~~~~~~~~~~~~ @@ -838,57 +926,6 @@ The following test checks for characters (`char`) are available: An automatically generated message is printed in case the test check fails. -Integers -~~~~~~~~ - -The following test checks for integers are available: - -.. code-block:: c - - void T_eq_xyz(I a, I e); - void T_assert_eq_xyz(I a, I e); - void T_quiet_eq_xyz(I a, I e); - void T_step_eq_xyz(unsigned int step, I a, I e); - void T_step_assert_eq_xyz(unsigned int step, I a, I e); - - void T_ne_xyz(I a, I e); - void T_assert_ne_xyz(I a, I e); - void T_quiet_ne_xyz(I a, I e); - void T_step_ne_xyz(unsigned int step, I a, I e); - void T_step_assert_ne_xyz(unsigned int step, I a, I e); - - void T_ge_xyz(I a, I e); - void T_assert_ge_xyz(I a, I e); - void T_quiet_ge_xyz(I a, I e); - void T_step_ge_xyz(unsigned int step, I a, I e); - void T_step_assert_ge_xyz(unsigned int step, I a, I e); - - void T_gt_xyz(I a, I e); - void T_assert_gt_xyz(I a, I e); - void T_quiet_gt_xyz(I a, I e); - void T_step_gt_xyz(unsigned int step, I a, I e); - void T_step_assert_gt_xyz(unsigned int step, I a, I e); - - void T_le_xyz(I a, I e); - void T_assert_le_xyz(I a, I e); - void T_quiet_le_xyz(I a, I e); - void T_step_le_xyz(unsigned int step, I a, I e); - void T_step_assert_le_xyz(unsigned int step, I a, I e); - - void T_lt_xyz(I a, I e); - void T_assert_lt_xyz(I a, I e); - void T_quiet_lt_xyz(I a, I e); - void T_step_lt_xyz(unsigned int step, I a, I e); - void T_step_assert_lt_xyz(unsigned int step, I a, I e); - -The type variant `xyz` must be `schar`, `uchar`, `short`, `ushort`, `int`, -`uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, -`i64`, `u64`, `iptr`, `uptr`, `ssz`, or `sz`. - -The type name `I` must be compatible to the type variant. - -An automatically generated message is printed in case the test check fails. - RTEMS Status Codes ~~~~~~~~~~~~~~~~~~ @@ -1028,6 +1065,35 @@ RTEMS, floating-point operations are only supported in special tasks and may be forbidden in interrupt context. The formatted output functions provided by the test framework work in every context. +Utility +------- + +You can stop a test case via the ``T_stop()`` function. This function does not +return. You can indicate unreachable code paths with the ``T_unreachable()`` +function. If this function is called, then the test case stops. + +You can busy wait with the ``T_busy()`` function: + +.. code-block:: c + + void T_busy(uint_fast32_t count); + +It performs a busy loop with the specified iteration count. This function is +optimized to not perform memory accesses and should have a small jitter. The +loop iterations have a processor-specific duration. + +You can get an iteration count for the ``T_busy()`` function which corresponds +roughly to one clock tick interval with the ``T_get_one_clock_tick_busy()`` +function: + +.. code-block:: c + + uint_fast32_t T_get_one_clock_tick_busy(void); + +This function requires a clock driver. It must be called from thread context +with interrupts enabled. It may return a different value each time it is +called. + Time Services ------------- @@ -1353,6 +1419,150 @@ reported. M:E:Empty:D:0.015188063 E:measure_empty:N:1:F:0:D:14.284869 +Interrupt Tests +--------------- + +In the operating system implementation you may have two kinds of critical +sections. Firstly, there are low-level critical sections protected by +interrupts disabled and maybe also some SMP spin lock. Secondly, there are +high-level critical sections which are protected by disabled thread +dispatching. The high-level critical sections may contain several low-level +critical sections. Between these low-level critical sections interrupts may +happen which could alter the code path taken in the high-level critical +section. + +The test framework provides support to write test cases for high-level critical +sections though the `T_interrupt_test()` function: + +.. code-block:: c + + typedef enum { + T_INTERRUPT_TEST_INITIAL, + T_INTERRUPT_TEST_ACTION, + T_INTERRUPT_TEST_BLOCKED, + T_INTERRUPT_TEST_CONTINUE, + T_INTERRUPT_TEST_DONE, + T_INTERRUPT_TEST_EARLY, + T_INTERRUPT_TEST_INTERRUPT, + T_INTERRUPT_TEST_LATE, + T_INTERRUPT_TEST_TIMEOUT + } T_interrupt_test_state; + + typedef struct { + void (*prepare)(void *arg); + void (*action)(void *arg); + T_interrupt_test_state (*interrupt)(void *arg); + void (*blocked)(void *arg); + uint32_t max_iteration_count; + } T_interrupt_test_config; + + T_interrupt_test_state T_interrupt_test( + const T_interrupt_test_config *config, + void *arg + ); + +This function returns ``T_INTERRUPT_TEST_DONE`` if the test condition was +satisfied within the maximum iteration count, otherwise it returns +``T_INTERRUPT_TEST_TIMEOUT``. The interrupt test run uses the specified +configuration and passes the specified argument to all configured handlers. +The function shall be called from thread context with interrupts enabled. + +.. image:: ../images/eng/interrupt-test.* + :scale: 60 + :align: center + +The interrupt test uses an *adaptive bisection algorithm* to try to hit the +code section under test by an interrupt. In each test iteration, it waits for +a time point one quarter of the clock tick interval after a clock tick using +the monotonic clock. Then it performs a busy wait using ``T_busy()`` with a +busy count controlled by the adaptive bisection algorithm. The test maintains +a sample set of upper and lower bound busy wait count values. Initially, the +lower bound values are zero and the upper bound values are set to a value +returned by ``T_get_one_clock_tick_busy()``. The busy wait count for an +iteration is set to the middle point between the arithmetic mean of the lower +and upper bound sample values. After the action handler returns, the set of +lower and upper bound sample values is updated based on the test state. If the +test state is ``T_INTERRUPT_TEST_EARLY``, then the oldest upper bound sample +value is replaced by the busy wait count used to delay the action and the +latest lower bound sample value is slightly decreased. Reducing the lower +bound helps to avoid a zero length interval between the upper and lower bounds. +If the test state is ``T_INTERRUPT_TEST_LATE``, then the oldest lower bound +sample value is replaced by the busy wait count used to delay the action and +the latest upper bound sample value is slightly increased. In all other test +states the timing values remain as is. Using the arithmetic mean of a sample +set dampens the effect of each test iteration and is an heuristic to mitigate +the influence of jitters in the action code execution. + +The optional *prepare* handler should prepare the system so that the *action* +handler can be called. It is called in a tight loop, so all the time consuming +setup should be done before ``T_interrupt_test()`` is called. During the +preparation the test state is ``T_INTERRUPT_TEST_INITIAL``. The preparation +handler shall not change the test state. + +The *action* handler should call the function which executes the code section +under test. The execution path up to the code section under test should have a +low jitter. Otherwise, the adaptive bisection algorithm may not find the right +spot. + +The *interrupt* handler should check if the test condition is satisfied or a +new iteration is necessary. This handler is called in interrupt context. It +shall return ``T_INTERRUPT_TEST_DONE`` if the test condition is satisfied and +the test run is done. It shall return ``T_INTERRUPT_TEST_EARLY`` if the +interrupt happened too early to satisfy the test condition. It shall return +``T_INTERRUPT_TEST_LATE`` if the interrupt happened too late to satisfy the +test condition. It shall return ``T_INTERRUPT_TEST_CONTINUE`` if the test +should continue with the current timing settings. Other states shall not be +returned. It is critical to return the early and late states if the test +condition was not satisfied, otherwise the adaptive bisection algorithm may not +work. The returned state is used to try to change the test state from +``T_INTERRUPT_TEST_ACTION`` to the returned state. + +The optional *blocked* handler is invoked if the executing thread blocks during +the action processing. It should remove the blocking condition of the thread +so that the next iteration can start. It can use +``T_interrupt_change_state()`` to change the interrupt test state. + +The *max iteration count* configuration member defines the maximum iteration +count of the test loop. If the maximum iteration count is reached before the +test condition is satisfied, then ``T_interrupt_test()`` returns +``T_INTERRUPT_TEST_TIMEOUT``. + +The *interrupt* and *blocked* handlers may be called in arbitrary test states. + +The *action*, *interrupt*, and *blocked* handlers can use +``T_interrupt_test_get_state()`` to get the current test state: + +.. code-block:: c + + T_interrupt_test_state T_interrupt_test_get_state(void); + +The *action*, *interrupt*, and *blocked* handlers can use +``T_interrupt_test_change_state()`` to try to change the test state from an +expected state to a desired state: + +.. code-block:: c + + T_interrupt_test_state T_interrupt_test_change_state( + T_interrupt_test_state expected_state, + T_interrupt_test_state desired_state + ); + +The function returns the previous state. If it **differs from the expected +state**, then the requested state **change to the desired state did not take +place**. In an SMP configuration, do not call this function in a tight loop. +It could lock up the test run. To busy wait for a state change, use +``T_interrupt_test_get_state()``. + +The *action* handler can use ``T_interrupt_test_busy_wait_for_interrupt()`` to +busy wait for the interrupt: + +.. code-block:: c + + void T_interrupt_test_busy_wait_for_interrupt(void); + +This is useful if the action code does not block to wait for the interrupt. If +the action handler just returns the test code immediately prepares the next +iteration and may miss an interrupt which happens too late. Test Runner ----------- diff --git a/images/eng/interrupt-test.odg b/images/eng/interrupt-test.odg new file mode 100644 index 0000000..6645fb2 Binary files /dev/null and b/images/eng/interrupt-test.odg differ diff --git a/images/eng/interrupt-test.pdf b/images/eng/interrupt-test.pdf new file mode 100644 index 0000000..fe8bc19 Binary files /dev/null and b/images/eng/interrupt-test.pdf differ diff --git a/images/eng/interrupt-test.png b/images/eng/interrupt-test.png new file mode 100644 index 0000000..4a4aca8 Binary files /dev/null and b/images/eng/interrupt-test.png differ -- cgit v1.2.3