user: Add RTEMS executable and test documentation.

17 files changed, 1371 insertions, 47 deletions

diff --git a/user/exe/debugging.rst b/user/exe/debugging.rst
new file mode 100644
index 0000000..3d1aa38
--- /dev/null
+++ b/user/exe/debugging.rst
@@ -0,0 +1,119 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Debugging
+=========
+.. index:: Debugging
+
+An RTEMS executable is debugged by loading the code, data and read-only data
+into a target with a debugger connected. The debugger running on a host
+computer accesses the ELF file reading the debug information it contains.
+
+The executable being debugged needs to be built with the compiler and linker
+debug options enabled. Debug information makes the ELF executable file large
+but it does not change the binary footprint of the executable when resident in
+the target. Target boot loaders and file conversion tools extract the binary
+code, data and read-only data to create the file embedded on the target.
+
+An ELF executable built with debug information contains DWARF debug
+information. DWARF is a detailed description of the executable a debugger uses
+to locate functions, find data, understand the type and structure of a
+variable, and know how much entry code every call has. The debugger uses this
+information to set breaks points, step functions, step instructions, view the
+data and much more.
+
+We recommend the compiler and linker debug options are always enabled. An ELF
+file with debug information can be used to investigate a crash report from a
+production system if the production ELF image is archived. The RTEMS tools
+chain provides tools that can take an address from a crash dump and find the
+corresponding instruction and source line. The extra size the debug information
+adds does not effect the target footprint and the extra size on a host is small
+compared to the benefits it brings.
+
+A desktop or server operating system's kernel hosts the executable being
+debugged handling the interaction with the executable and the debugger. The
+debugger knows how to communicate to the kernel to get the information it
+needs. Debugging an embedded executable needs an extra piece called an agent to
+connect the target to the debugger. The agent provides a standard remote interface to
+the debugger and an agent specific connection to the target.
+
+.. _fig-exe-debug:
+
+.. figure:: ../../images/user/exe-debug.png
+   :width: 80%
+   :alt: Embedded Executable Debugging
+   :figclass: align-center
+
+   Embedded Executable Debugging
+
+The RTEMS tool chain provides the GNU debugger GDB. GDB has a remote protocol
+that can run over networks using TCP and UDP protocols. The GDB remote protocol
+is available in a number of open source and commercial debugging
+solutions. Network debugging using the remote protocol helps setting up a
+laboratory, the targets can be remote from the developers desktop allowing for
+better control of the target hardware while avoiding the need to plug devices
+in to an expensive desktop or server machine.
+
+The following are some examples of GDB and GDB server environments RTEMS
+supports.
+
+.. index:: QEMU
+
+QEMU contains a debugging agent for the target being simulated. A QEMU command
+line option enables a GDB server and the simulator manages the interaction with
+the target processor and it's memory and caches.
+
+.. _fig-exe-debug-qemu:
+
+.. figure:: ../../images/user/exe-debug-qemu.png
+   :width: 70%
+   :alt: QEMU Executable Debugging
+   :figclass: align-center
+
+   QEMU Executable Debugging
+
+.. index:: OpenOCD
+.. index:: JTAG
+
+OpenOCD is a JTAG debugging package that interfaces to a wide of JTAG
+pods. JTAG is a low level high speed serial interface modern processors provide
+as a means of controlling the core processing logic. The features available depend on
+the architecture and processor. Typical functions include:
+
+#. Processor control and register access
+
+#. System level register access to allow SOC initialization
+
+#. General address space access
+
+#. Cache and MMU control
+
+#. Break and watch points
+
+.. _fig-exe-debug-qemu:
+
+.. figure:: ../../images/user/exe-debug-jtag.png
+   :width: 70%
+   :alt: OpenOCD JTAG Executable Debugging
+   :figclass: align-center
+
+   OpenOCD JTAG Executable Debugging
+
+.. index:: libdebugger
+
+The RTEMS kernel has a debugging agent called ``libdebugger``. This is a
+software based agent that runs within RTEMS using network services to provide a
+remote GDB protocol interface. A growing number of architectures are
+supported. The RTEMS debugging agent is for application development providing
+thread aware stop model debug experience.
+
+.. _fig-exe-debug-libdebugger:
+
+.. figure:: ../../images/user/exe-debug-libdebugger.png
+   :width: 70%
+   :alt: Libdebugger Executable Debugging
+   :figclass: align-center
+
+   Libdebugger Executable Debugging
diff --git a/user/exe/executables.rst b/user/exe/executables.rst
new file mode 100644
index 0000000..d911006
--- /dev/null
+++ b/user/exe/executables.rst
@@ -0,0 +1,102 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+RTEMS Executable
+================
+.. index:: RTEMS Executable
+
+Running executables is the most important part of working with RTEMS, it is
+after all how you run your application and use the RTEMS kernel services.
+
+An RTEMS executable is embedded in a target and executing an embedded
+executable has challenges not faced when executing software on a desktop or
+server computer. A desktop or server operating system kernel provides all the
+support needed to bring an executable's code and data into a process's address
+space passing control to it and cleaning up when it exits. An embedded target
+has to provide similar functionality to execute an embedded executable.
+
+An RTEMS Source Builder (RSB) built RTEMS tool chain is used to create RTEMS
+executables. The tool chain executable creates a fixed position statically
+linked Extendable Loader Format (ELF) file that contains the RTEMS kernel,
+standard libraries, 3rd party libraries and application code. RTEMS executes in
+a single address space which means it does not support the ``fork`` or ``exec``
+system calls so statically linking all the code is the easiest and best way to
+create an executable.
+
+An RTEMS application is constructed vertically with the RTEMS kernel, BSP
+support code and drivers close to the hardware, above which sit the RTEMS
+Application Programming Interfaces (API) for control of threads, mutex and
+other resources an application may use. Middle-ware services like networking,
+interpreted languages, and protocol stacks sit between the RTEMS APIs and the
+application components. The software built into an executable can be see as a
+vertical software stack.
+
+.. _fig-exe-vert-stack:
+
+.. figure:: ../../images/user/exe-vert-stack.png
+   :width: 35%
+   :alt: Vertical Software Stack
+   :figclass: align-center
+
+   Vertical Software Stack
+
+Building an Application
+=======================
+.. index:: Building an Application
+
+RTEMS views any code it is running and using it's interfaces as an
+application. RTEMS conforms to a number of international standards such as
+POSIX and can build and run portable code written in languages such as C, C++
+and Ada.
+
+Applications are built from source into ELF object files, 3rd party packages
+can be built as libraries or they can be imported as source into an application
+code base. The application, 3rd party packages, RTEMS and standard libraries
+are linked to create the RTEMS executable. The executable is transferred to the
+target and a bootloader loads it from the non-volatile storage into RAM or the
+code is executed in place in the non-volatile storage. The target hardware
+defines what happens.
+
+.. _fig-exe-app:
+
+.. figure:: ../../images/user/exe-app.png
+   :width: 90%
+   :alt: Building an Application
+   :figclass: align-center
+
+   Building an Application
+
+The standard and 3rd party libraries are a collection of object files built
+using the same set of tools the application source is compiled with. The
+package collects it's object files into an archive or library.
+
+RTEMS does not provide a standard application build system. The RTEMS ecosystem
+provides support so a range of build systems can be used. Applications can be
+built with ``make``, ``autotools``, ``cmake``, ``waf`` and more. User should
+select a build system that meets their project, system, corporate or personal
+needs.
+
+Machine Flags and ABI
+---------------------
+.. index:: Machine flags
+.. index:: Application Binary Interface
+.. index:: ABI
+
+
+All code in an RTEMS executable must be built with the same machine flags. The
+machine flags control the instruction set and application binary interface
+(ABI) the compiler generates. As the executable is statically linked all code
+must use the same instruction set the hardware is configured to support and all
+code must conform to the same ABI. Any variation can result in unpredictable
+behavior such as crashes, failures or lock ups. It is recommend an executable
+is built with the same or equivalent tool set. Mixing of tool set versions can
+also result in undefined behavior. The RTEMS tool ``rtems-execinfo`` can audit
+an RTEMS executable and list the machine flags and compilers used.
+
+RTEMS by default does not support instruction emulation for unsupported
+instructions. RTEMS applications are normally built from source so binary
+compatibility is not as important as performance. Instruction emulation is
+costly to execute and rebuilding the executable with the correct instruction
+set only needs to be done once.
diff --git a/user/exe/execution.rst b/user/exe/execution.rst
new file mode 100644
index 0000000..029bd24
--- /dev/null
+++ b/user/exe/execution.rst
@@ -0,0 +1,49 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Target Execution
+================
+.. index::  Target Execution
+
+Fixed position statically linked executables have a fixed address in a target's
+address space. The location in the address space for code, data and read-only
+data is fixed. The BSP defines the memory map and it is set by the BSP
+developer based on the target's hardware requirements and it's bootloader.
+
+Targets typically contains a bootloader that is executed after the target's
+processor exits reset. A bootloader is specific to a target's processor and
+hardware configuration and is responsible for the low level initialization of
+the hardware resources needed to load and execute an operating system's
+kernel. In the case of RTEMS this is the RTEMS executable.
+
+Bootloaders vary in size, complexity and functionality. Some architectures have
+a number of bootloader stages and others have only minimal support. An example
+of a high end system is Xilinx's Zynq processor with three stages. First a mask
+ROM in the System On Chip (SOC) executes after reset loading a first stage
+bootloader (FSBL) from an SD card, QSPI flash or NAND flash depending on
+signals connected to the device. The FSBL loads a second stage bootloader
+(SSBL) such as U-Boot and this loads the kernel. U-Boot can be configured to
+load a kernel from a range of media and file system formats as well as over a
+network using a number of protocols. This structure provides flexibility at the
+system level to support development environments such as a workshop or
+laboratory through to tightly control production configurations.
+
+Bootloaders often have custom formats for the executable image they load. The
+formats can be simple to keep the bootloader simple or complex to support
+check-sums, encryption or redundancy in case an image becomes corrupted. A
+bootloader often provides a host tool that creates the required file from the
+RTEMS executable's ELF file.
+
+If RTEMS is to run from RAM the bootloader reads the image and loads the code,
+initialized data and read-only data into the RAM and then jumps to a known
+entry point. If the code is executed from non-volatile storage the process to
+write the image into that storage will have extracted the various binary parts
+and written those to the correct location.
+
+The important point to note is the binary parts of the executable are somehow
+loaded into the target's address space ready to execute. The way this done may
+vary but the out come is always the same, the binary code, data and read-only
+data is resident in the processor's address space at the BSP defined
+addresses.
diff --git a/user/exe/index.rst b/user/exe/index.rst
new file mode 100644
index 0000000..c40b773
--- /dev/null
+++ b/user/exe/index.rst
@@ -0,0 +1,19 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Executables
+***********
+.. index:: Executable
+.. index:: Embedded executable
+
+This section discusses what an RTEMS executable is and what happens when you
+execute it in a target. The section discusses how an application executable is
+created, what happens when an executable is loaded and run as well as
+debugging an execiutable.
+
+.. include:: executables.rst
+.. include:: execution.rst
+.. include:: initialization.rst
+.. include:: debugging.rst
diff --git a/user/exe/initialization.rst b/user/exe/initialization.rst
new file mode 100644
index 0000000..9ecee16
--- /dev/null
+++ b/user/exe/initialization.rst
@@ -0,0 +1,130 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+BSP Initialization
+==================
+.. index:: BSP Initialization
+
+The bootloader jumps or calls the RTEMS executable's entry point, normally a
+fixed address. The BSP entry point or start up code performs:
+
+#. Low level processor specific initialization that such as setting control
+   registers so the processor is operating in a mode RTEMS is built for
+
+#. Cache flushing, clearing and invalidation
+
+#. Memory management unit (MMU) set up if required
+
+#. Clear the uninitialized data section
+
+#. Process a command line if supported by the bootloader
+
+#. Call ``bootcard`` which disabled interrupts, saves away a command line if
+   the BSP supports it then call the RTEMS kernel early initialize entry point
+   ``rtems_initialize_executive``. This call never returns.
+
+Further BSP initialization happens as part of RTEMS kernel's System
+Initialization process. The following handlers are declared and if provided are
+placed at the beginning of the initialization handler list. The BSP can
+provides:
+
+``bsp_work_area_initialize``
+  This function determines the amount of memory that can be given to RTEMS for
+  the workspace and the C library heap which ``malloc`` uses. The call
+  typically uses the ``bsp_work_area_initialize_default`` to perform actually
+  perform the initialization.
+
+``bsp_start``
+  This function is specialized for each architecture and even for some BSPs. It
+  performs the low level initialization RTEMS needs so it can run on the
+  architecture and BSP.
+
+``bsp_predriver_hook``
+  This function can be used to initialize hardware drivers depend on such as
+  configuring an interrupt controller. The default version is empty and does
+  nothing.
+
+BSPs all perform similar operations with common functionality and the RTEMS
+kernel provides common code that can be shared between BSPs. The use of the
+common code is encouraged for all new BSPs.
+
+RTEMS Initialization
+====================
+.. index:: RTEMS Initialization
+
+The RTEMS kernel initialization is:
+
+#. Invoke the registered system initialization handlers
+
+#. Set the system state to **up**
+
+#. If the kernel supports SMP request multitasking start. All online cores are
+   transferred to the **ready to start multitasking** state.
+
+#. Start threaded multitasking. RTEMS starts multitasking by getting the first
+   thread to run and dispatching it.
+
+C++ static object constructors are called in the context of the first running
+thread before the thread body is entered.
+
+System Initialization Handlers
+------------------------------
+
+RTEMS supports the automatic registration of services used in
+applications. This method of initialization automatically configures RTEMS with
+only the services used in an application. There is no manual configuration of
+services used and no updating of initialization function tables.
+
+RTEMS uses specialized sections in the ELF executable to perform this task. The
+system is based on the `FreeBSD SYSINT Framework
+<https://www.freebsd.org/doc/en/books/arch-handbook/sysinit.html>`_. Ordered
+initialization is performed before multitasking is started.
+
+The RTEMS Tool ``rtems-exeinfo`` can provide some detail about the registered
+handlers. The following shows the initialization handlers for the *hello world*
+sample application in the RTEMS kernel's testsuite::
+
+ $ rtems-exeinfo --init arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe
+ RTEMS Executable Info 5.5416cfa39dd6
+  rtems-exeinfo --init arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe
+ exe: arm-rtems5/c/xilinx_zynq_zedboard/testsuites/samples/hello.exe
+
+ Compilation:
+  Producers: 2
+   |  GNU AS 2.31.1: 14 objects
+   |  GNU C11 7.3.0 20180125 (RTEMS 5, RSB e55769c64cf1a201588565a5662deafe3f1ccdcc, Newlib 103b055035fea328f8bc7826801760fb1c055683): 284 objects
+  Common flags: 4
+   | -march=armv7-a -mthumb -mfpu=neon -mfloat-abi=hard
+
+ Init sections: 2
+  .init_array
+   0x001047c1 frame_dummy
+  .rtemsroset
+   0x00104c05 bsp_work_area_initialize
+   0x00104c41 bsp_start
+   0x0010eb45 zynq_debug_console_init
+   0x0010ec19 rtems_counter_sysinit
+   0x0010b779 _User_extensions_Handler_initialization
+   0x0010c66d rtems_initialize_data_structures
+   0x00107751 _RTEMS_tasks_Manager_initialization
+   0x0010d4f5 _POSIX_Keys_Manager_initialization
+   0x0010dd09 _Thread_Create_idle
+   0x0010cf01 rtems_libio_init
+   0x001053a5 rtems_filesystem_initialize
+   0x0010546d _Console_simple_Initialize
+   0x0010c715 _IO_Initialize_all_drivers
+   0x001076d5 _RTEMS_tasks_Initialize_user_tasks_body
+   0x0010cfa9 rtems_libio_post_driver
+
+The section ``.rtemsroset`` lists the handlers called in order. The handlers
+can be split into the BSP initialization handlers that start the BSP:
+
+- ``bsp_work_area_initialize``
+- ``bsp_start``
+- ``zynq_debug_console_init``
+- ``rtems_counter_sysinit``
+
+And the remainder are handlers for services used by the application. The list
+varies based on the services the application uses.
diff --git a/user/index.rst b/user/index.rst
index 48870a1..0f60166 100644
--- a/user/index.rst
+++ b/user/index.rst
@@ -51,10 +51,12 @@ to the Community Project hosted at http://www.rtems.org/.
 	hardware/index
 	bsps/index
 
-	tools/index
-
+	exe/index
+        testing/index
         tracing/index
 
+	tools/index
+
 	support/index
 
 	glossary/index
diff --git a/user/test/create.rst b/user/test/create.rst
deleted file mode 100644
index b86453a..0000000
--- a/user/test/create.rst
+++ /dev/null
@@ -1,7 +0,0 @@
-.. comment SPDX-License-Identifier: CC-BY-SA-4.0
-
-Creating A Test
-===============
-.. index:: Creating a Test
-
-XXX: How to create a test.
diff --git a/user/test/index.rst b/user/test/index.rst
deleted file mode 100644
index 7ae43fe..0000000
--- a/user/test/index.rst
+++ /dev/null
@@ -1,10 +0,0 @@
-.. comment SPDX-License-Identifier: CC-BY-SA-4.0
-
-Test Suite
-**********
-
-XXX: All about the test suite.
-
-
-.. include:: running.rst
-.. include:: create.rst
diff --git a/user/test/running.rst b/user/test/running.rst
deleted file mode 100644
index 5184ae1..0000000
--- a/user/test/running.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-.. comment SPDX-License-Identifier: CC-BY-SA-4.0
-
-Running
-========
-.. index:: Running a Test
-
-XXX: How to run tests via waf.
-
-
-Host
-----
-
-XXX: Host-based tests
-
-
-Simulation
-----------
-.. index:: Test Simulation
-
-XXX: Simulator
-
-
-
-Hardware
---------
-XXX: Running on real hardware.
diff --git a/user/testing/configuration.rst b/user/testing/configuration.rst
new file mode 100644
index 0000000..a58b373
--- /dev/null
+++ b/user/testing/configuration.rst
@@ -0,0 +1,312 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Tester Configuration
+--------------------
+
+The RTEMS Tester and RTEMS Run are controlled by configuration data and
+scripts. The user specifies a BSP on the command line using the ``--rtems-bsp``
+option as well as optionally specifying a user configuration file using
+``--user-config``.
+
+The Figure :ref:`fig-tester-config-1` shows the various sources of
+configuration data and their format. The ``ini`` files are the standard INI
+format, the ``mc`` are the internal RTEMS Toolkit's Macro format, and ``cfg``
+is the RTEMS Toolkit's Configuration script format, the same format used by the
+RTEMS Source Builder.
+
+.. _fig-tester-config-1:
+
+.. figure:: ../../images/user/test-cfg-1.png
+   :width: 50%
+   :alt: RTEMS Tester and Run Configuration Files
+   :figclass: align-center
+
+   RTEMS Tester and Run Configuration Files
+
+Configuration data is held in a macro database keyed on the macro name. Macros
+can be expanded in configuration scripts using the syntax ``%{name}``. The
+macro database is layered using maps. The defaults and values created when a
+configure script runs live the in the ``global`` map. Values read from the BSP
+and User INI configuration files are loaded into maps based on the BSP
+name. This lets a single User configuration file contain specialized
+configuration values for a number of BSPs and the tester and run commands
+select the values based on the selected BSP. Macros are expanded using the BSP
+map first giving those values the highest priority. User defined values are
+loaded after the BSP configuration values overwriting them letting a user
+speckles a BSP's default configuration for their local needs.
+
+Figure :ref:`fig-tester-config-2` shows the configuration loading and script
+execution order.
+
+.. _fig-tester-config-2:
+
+.. figure:: ../../images/user/test-cfg-2.png
+   :width: 50%
+   :alt: RTEMS Tester and Run Configuration Load and Execute Sequence
+   :figclass: align-center
+
+   RTEMS Tester and Run Configuration Load and Execute Sequence
+
+Defaults
+^^^^^^^^
+
+The RTEMS Tester and RTEMS Run are primed using defaults from the file
+``rtems/testing/testing.mc``. All default settings can be overridden in a BSP or
+User configuration file.
+
+.. index:: BSP configuration, User configuration
+BSP and User Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The BSP and User configuration files are INI format files. The BSP
+configuration file has to have an INI section that is the name of the BSP
+passed on the command line. The section has the following mandatory values:
+
+.. index:: bsp
+``bsp``
+  The name of the BSP. The BSP name is used to create a macro map to hold the
+  BSP's configuration data. Typically this is the same as the BSP name used on
+  the command line.
+
+.. index:: arch
+``arch``
+  The name of the BSP architecture. This is need for the GDB configuration
+  scripts where the architecture specific GDB needs to run. It is mandatory so
+  the *arch/bsp* standard RTEMS BSP string can be used.
+
+.. index:: tester
+``tester``
+  The tester or run configuration script. This is the name of the configuration
+  script the RTEMS Tester or RTEMS Run executes as a back end. The ``tester``
+  value is typically of the form ``%{_rtscripts}/<script>`` where ``<script>``
+  is name of the back end script to be run.
+
+Target commands support expansion of specific tags to provide a convenient way
+for users to customize a local test environment. The parameters expanded are:
+
+.. index:: @ARCH@
+``@ARCH@``
+  The BSP architecture.
+
+.. index:: @BSP@
+``@BSP@``
+  The BSP's name set by the ``bsp`` value.
+
+.. index:: @EXE@
+``@EXE@``
+  The executable name as an absolute path
+
+.. index:: @FEXE@
+``@FEXE@``
+  The filtered executable if a ``target_exe_filter`` is provided else the
+  executable's file name.
+
+The following are optional and depend on the back end being used and the local
+target hardware set up:
+
+.. index:: jobs
+``jobs``
+  The jobs value sets the number of jobs that can be run at once. This setting
+  only effects the RTEMS Tester. The tester can run up to the ``jobs`` value of
+  tests concurrently. If the tester back end is a simulator running a job on
+  each available core lowers the total test time. Overloading a machine with
+  too many simulators running in parallel can slow down each simulation and
+  test timeouts may be recorded.
+
+.. index:: bsp_tty_dev
+``bsp_tty_dev``
+  The BSP's tty device. This can be a real device on the host machine the
+  executable is being run from or it can be a telnet server and port defined
+  using the stand host format. See :ref:`tester-consoles` for details.
+
+.. index:: target_pretest_command
+``target_pretest_command``
+  The pre-test command is a host shell command that is called before each test
+  runs. It can be used to construct a suitable environment or image needed by a
+  simulator or target. The RTEMS executate being run is provided as an argument
+  and the bootloader specific format is the output.
+
+ .. index:: target_posttest_command
+``target_posttest_command``
+  The post-test command is a host shell command that is called after each test
+  has finished. It can be used to destroy any environment or image created by
+  the pre-test command.
+
+.. index:: target_exe_filter
+``target_exe_filter``
+  The target executable filter transforms the executable name into a filtered
+  executable name. This filter lets the tester or run command track the name of
+  any generated file a pre-test command may generate. The syntax is a simplified
+  ``sed`` regular expression. The first character is a delimiter and there must
+  be 2 sections therefore 3 delimiter. The first section is a Python regular
+  expression and the second section is plain text that replaces anywhere the
+  regular expression matches. For example ``/\.exe/.exe.img/`` will search for
+  ``.exe`` in the executable name and replace it with ``.exe.img``. Note, there
+  is no need to escape the text in the second part, it is just plain test.
+
+.. index:: test_restarts
+``test_restarts``
+  The number of restarts before the test is considered ``invalid``. Currently
+  not used.
+
+.. index:: target_reset_regex
+``target_reset_regex``
+  The target reset regular expression. This is a `Python regular expression
+  <https://docs.python.org/2/library/re.html#regular-expression-syntax>`_ used
+  to filter the console input. If a match is made something has happened during
+  the boot process that requires a reset. The ``target_reset_command`` is
+  issued to perform the reset. Typically this field looks for boot loader error
+  messages that indicate the boot process as failed.
+
+.. index:: target_start_regex
+``target_start_regex``
+
+  The target start regular expression. This is a Python regular expression to
+  filter the console input to asynchronously detect if a target has reset. If a
+  board crashes running a test or at any point reset this filter detects the
+  restart and ends the test with a suitable result.
+
+.. index:: target_on_command
+``target_on_command``
+  The target on command is a host shell command that is called before the first
+  test. This command powers on a target. Targets should be left powered off
+  when not running tests or the target may request TFTP downloads that are for
+  another target interfering with those test results. We recommend you
+  implement this command as a target off command, a pause, then a target on
+  command.
+
+.. index:: target_off_command
+``target_off_command``
+  The target off command is a host shell command that is called after the last
+  test powering off the target.
+
+.. index:: target_reset_command
+``target_reset_command``
+  The target reset command is a host shell command that is called when the
+  target needs to be reset. This command can power cycle the target or toggle a
+  reset signal connected to the target. If you are power cycling a target make
+  sure you have a suitable pause to let the target completely power down.
+
+.. _tester-config-scripts:
+
+Configuration Scripts
+^^^^^^^^^^^^^^^^^^^^^
+
+Configuration scripts are provided for each supported RTEMS Tester and RTEMS
+Run back end and console management. The scripts are in the standard RTEMS
+Toolkit Configuration Script format. Please refer to the RTEMS Source Builder
+documentation for the basic scripting syntax and usage.
+
+The RTEMS Tester and RTEMS Run specializes the standard configuration syntax
+providing a directive for the console and each supported back end. The
+supported directives are:
+
+- ``%console``
+- ``%execute``
+- ``%gdb``
+- ``%tftp``
+
+.. _tester-config-console:
+.. index:: Console, %console
+
+Console
+~~~~~~~
+
+The ``%console`` configures the console used to access the target's
+console. The console can be a process's ``stdout``, a termios tty on Unix and
+MacOS and Telnet on all hosts. The directive accepts:
+
+``stdio``
+  The standard output stream from the executing processing.
+
+``tty <dev> <settings>``
+  The name of the ``tty`` to open and use. The ``tty`` device or ``<dev>`` can
+  be a *termio* device and the ``<settings>`` are standard termios values.
+
+  The Python termios document provides details of the settings that can be
+  controlled. The settings are a single string where prefix the value with
+  ``~`` negates the setting. Setting are:
+
+   - ``B115200`` (an example buadrate)
+   - ``BRKINT``
+   - ``IGNBRK``
+   - ``IGNCR``
+   - ``ICANON``
+   - ``ISIG``
+   - ``IEXTEN``
+   - ``ECHO``
+   - ``CLOCAL``
+   - ``CRTSCTS``
+   - ``VMIN=<value>``
+   - ``VTIME=<value``
+
+A example in a configuration script is::
+
+  %define bsp_tty_dev      /dev/ttyUSB2
+  %define bsp_tty_settings B115200,~BRKINT,IGNBRK,IGNCR,~ICANON,~ISIG,~IEXTEN,~ECHO,CLOCAL,~CRTSCTS,VMIN=1,VTIME=2
+
+A example BSP or User configuration file is::
+
+  [bsp-special]
+  bsp              = example-bsp
+  bsp_tty_dev      = /dev/ttyUSB2
+  bsp_tty_settings = B115200,~BRKINT,IGNBRK,IGNCR,~ICANON,~ISIG,~IEXTEN,~ECHO,CLOCAL,~CRTSCTS,VMIN=1,VTIME=2
+
+The console directive is managed in the ``%{_rtscripts}/console.cfg``
+configuration script. If the ``%{console_stdio}`` is defined the console will
+be ``stdio`` else the console will be the BSP console or ``%{bsp_tty_dev}``.
+
+Telnet can be combined with the ``ser2net`` daemon to remotely access a
+target's physical serial UART interface.
+
+.. _tester-config-execute:
+.. index:: Execute, %execute
+
+Execute
+~~~~~~~
+
+The ``%execute`` directive executes a command for each rest. The execute forks
+the command and arguments supplied to the execute directive and captures the
+``stdout`` stream as the console. If the console directive is set to ``stdout``
+the sub-processes ``stdout`` stream is used as the console.
+
+The RTEMS Tester will run parallel tests as jobs.
+
+An example is::
+
+  %execute %{run_cmd} %{run_opts} %{test_executable} %{test_executable_opts}
+
+.. _tester-config-gdb:
+.. index:: GDB, %gdb
+
+GDB
+~~~
+
+The ``%gdb`` directive executes GDB in the machine interface mode give the
+RTEMS Tester and RTEMS Run commands control. The console is taken from
+GDB if it is ``stdout``.
+
+The RTEMS Tester will run parallel tests as jobs.
+
+An example is::
+
+  %gdb %{gdb_cmd} %{test_executable} %{gdb_script}
+
+.. _tester-config-tftp:
+.. index:: TFTP, %tftp
+
+TFTP
+~~~~
+
+The ``%tftp`` directive starts a TFTP session on a specified port sending the
+test executable to the target over a networking using the TFTP protocol.
+
+The RTEMS Tester will run only one test at a time. There is just one physical
+board running the test.
+
+An example is::
+
+  %tftp %{test_executable} %{tftp_port}
diff --git a/user/testing/consoles.rst b/user/testing/consoles.rst
new file mode 100644
index 0000000..9fa84ee
--- /dev/null
+++ b/user/testing/consoles.rst
@@ -0,0 +1,66 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+.. _tester-consoles:
+
+Consoles
+--------
+
+The RTEMS Tester uses the target's console output to determine the state of a
+test. Console interfaces vary depending on the testing mode, the BSP, and the
+target hardware.
+
+Consoles for simulator work best if mapped to the simulator's ``stdout``
+interface. The RTEMS Tester can capture and process the ``stdout`` data from a
+simulator while it is running.
+
+Target hardware console interfaces can vary. The most universal and stable
+interface target hardware is a UART interface. There are a number of physical
+interfaces for UART data these days. They are:
+
+#. RS232
+
+#. TTL
+
+#. USB
+
+RS232 is still present on a number of targets. The best solution is to use a
+RS232 to USB pod and convert the port to USB.
+
+TTL is common on a number of boards where cost is important. A console
+interface is typically a development tool and removing the extra devices need
+to convert the signal to RS232 or directly to USB is not needed on production
+builds of the target. There is a standard header pin out for TTL UART consoles
+and you can purchase low cost cables with the header and a built in UART to USB
+converter. The cables come is different voltage levels so make sure you check
+and use the correct voltage level.
+
+The USB interface on a target is typcially a slave or OTG interface and all you
+need to a standard USB cable.
+
+We recommend a low cost and low power device to be a terminal server. A
+Raspberry Pi or similar low cost computer running Linux can be set up quickly
+and with a powered USB hub and can support a number of USB UART ports. A USB
+hub with a high power port is recommended that can suppy the Raspberry Pi.
+
+The open source daemon ``ser2net`` is easy to configure to map the USB UART
+ports to the Telnet protocol. There is no need for security because a typical
+test environment is part of a lab network that should be partitioned off from
+an enginnering or corportate network and not directly connected to the
+internet.
+
+A test set up like this lets you place a terminal server close to your target
+hardware providing you with the flexibility to select where you run the RTEMS
+Tester. It could be your desktop or an expensive fast host machine in a server
+rack. None of this equipment needs to directly interface to the target
+hardware.
+
+The RTEMS Tester directly supports the telnet protcol as a console and can
+interface to the ``ser1net`` server. The telnet console will poll the server
+waiting for the remote port to connect. If the terminal server ``ser2net`` does
+not have a ``tty`` device it will not listen on the port assigned to that
+``tty``. A USB ``tty`` can come and go depending on the power state of the
+hardware and the target hardware's design and this can cause timing issues if
+the target hardware is power cycled as part of a reset process.
diff --git a/user/testing/gdb-jtag.rst b/user/testing/gdb-jtag.rst
new file mode 100644
index 0000000..367d1a2
--- /dev/null
+++ b/user/testing/gdb-jtag.rst
@@ -0,0 +1,28 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+GDB and JTAG
+------------
+.. index:: GDB, JTAG, Testing
+
+GDB with JTAG provides a low level way to runs tests on hardware with limited
+resources. The RTEMS Tester runs and controls an instance of GDB per test and
+GDB connects via the GDB remote protocol to a GDB server that interfaces to the
+JTAG port of a target.
+
+.. _fig-tester-gdb-jtag:
+
+.. figure:: ../../images/user/test-gdb-jtag.png
+   :width: 35%
+   :alt: RTEMS Tester using GDB and  JTAG
+   :figclass: align-center
+
+   RTEMS Tester using GDB and JTAG
+
+The :ref:`fig-tester-gdb-jtag` figure shows the structure of RTEMS Testing
+using GDB and JTAG. The executables are built and the ``rtems-test`` command is
+run from the top of the build directory. The RTEMS Tester executes the BSP
+architecture's GDB and expects the user to provide a ``gdb-script`` to connect
+t the JTAG GDB server.
diff --git a/user/testing/index.rst b/user/testing/index.rst
new file mode 100644
index 0000000..f3a6846
--- /dev/null
+++ b/user/testing/index.rst
@@ -0,0 +1,46 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Testing
+*******
+
+RTEMS developers run test executables when adding new features or testing a bug
+fix. All tests are run to make sure changes do not introduce regressions. Users
+can run the RTEMS tests to be certain the build of the kernel they have is
+functioning.
+
+The section describes using and configuring the RTEMS Tester and RTEMS Run
+tools, the types of laboratory set ups supported and how to add your BSP to the
+framework. The tools command line interfaces are detailed in
+:ref:`rtems-tester-command`.
+
+An RTEMS Test is an RTEMS executable where the application code is a
+test. Tests in RTEMS print banners to the console to indicate the configuration
+of the test and if it has start and finished.
+
+The RTEMS Tools Project provides the RTEMS Tester and RTEMS Run tools. The
+RTEMS Tester command is ``rtems-test`` and the RTEMS Run command is
+``rtems-run``. These commands manage the complexity of running embedded
+executables. The commands provide a consistent command line interface to a
+testing framework that supports the various run time and testing scenarios we
+encounter such as simulators, GDB and executing directly on target hardware.
+
+The RTEMS kernel code contains an extensive set of tests to exercise and test
+the RTEMS kernel. The tests check functionality, provide coverage testing and
+make sure the kernel is operating as intended on your target system. The
+testsuite has support to make creating a test simple and uniform.
+
+The tests are built by adding ``--enable-tests`` to the RTEMS build
+configuration command line. There are over 600 tests and building them does
+extend the RTEMS kernel's build time and use more disk space but it worth
+building and running them. The RTEMS test executables have the ``.exe`` file
+extension.
+
+.. include:: tests.rst
+.. include:: configuration.rst
+.. include:: consoles.rst
+.. include:: simulation.rst
+.. include:: gdb-jtag.rst
+.. include:: tftp.rst
diff --git a/user/testing/simulation.rst b/user/testing/simulation.rst
new file mode 100644
index 0000000..865673f
--- /dev/null
+++ b/user/testing/simulation.rst
@@ -0,0 +1,27 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Simulation
+----------
+.. index:: Simulation, Testing
+
+Simulation is a important regression and development tool for RTEMS. Developers
+use simulation to work on core parts of RTEMS as it provides excellent
+debugging supporting. Simulation run via the RTEMS Tester allows a test to run
+on each core of your testing host machine lower the time to run all tests.
+
+.. _fig-tester-simulation:
+
+.. figure:: ../../images/user/test-simulation.png
+   :width: 30%
+   :alt: RTEMS Tester Simulation
+   :figclass: align-center
+
+   RTEMS Tester Simulation
+
+The :ref:`fig-tester-simulation` figure shows the structure of RTEMS Testing
+using simulation. The executables are built and the ``rtems-test`` command is
+run from the top of the build directory. The RTEMS Tester executes the
+BSP specific simulator for each test capturing the output
diff --git a/user/testing/tests.rst b/user/testing/tests.rst
new file mode 100644
index 0000000..345e9c0
--- /dev/null
+++ b/user/testing/tests.rst
@@ -0,0 +1,210 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+Test Banners
+------------
+
+All test output banners or strings are embedded in each test and the test
+outputs them to the BSP's console as it executes. The RTEMS Tester captures the
+BSP's console and uses this information to manage the state of the executing
+test. The banner strings are:
+
+.. _test-banner-begin:
+.. index:: test begin, TEST BEGIN
+
+``*** BEGIN TEST <name> ***``
+  The test has loaded, RTEMS has initialized and the test specific code is
+  about to start executing. The ``<name>`` field is the name of the test. The
+  test name is internal to the test and may not match the name of the
+  executable. The test name is informative and not used by the RTEMS Tester.
+
+.. _test-banner-end:
+.. index:: test end, TEST END
+
+``*** END TEST <name> ***``
+  The test has finished without error and has passed. The ``<name>`` field is
+  the name of the test. See the :ref:`Test Begin Banner <test-banner-begin>`
+  for details about the name.
+
+.. index:: test banner version, TEST VERSION
+
+``*** TEST VERSION: <version>``
+  The test prints the RTEMS version return by the RTEMS Version API as
+  ``<version>``. All tests must match the first test's version or the Wrong
+  Version error count is incremented.
+
+.. _test-banner-state:
+.. index:: test state, TEST STATE
+
+``*** TEST STATE: <state>``
+  The test is tagged in the RTEMS sources with a special ``<state>`` for this
+  BSP. See :ref:`Test States <test-states>` for the list of possible
+  states. The state banner lets the RTEMS Tester categorize and manage the
+  test. For example a user input test typically needing user interaction may
+  never complete producing an *invalid* test result. A user input test is
+  terminated to avoid extended delays in a long test run.
+
+.. _test-banner-build:
+.. index:: test build, TEST BUILD
+
+``*** TEST BUILD: <build>``
+  The test prints the RTEMS build as a space separated series of labels as
+  ``<build>``. The build labels are created from the configuration settings in
+  the Super Score header file ``rtems/score/cputops.h``. All tests must match
+  the first test's build or the Wrong Build error count is incremented.
+
+.. _test-banner-tools:
+.. index:: test tools, TEST TOOLS
+
+``*** TEST TOOLS: <version>``
+  The test prints the RTEMS tools version returned the GGC internal macro
+  ``_VERSION_`` as ``<version>``. All tests must match the first test's tools
+  version string or the Wrong Tools error count is incremented.
+
+.. _test-states:
+.. index:: Test states
+
+Test States
+-----------
+
+The tests states are:
+
+.. index:: test state passed
+``passed``
+  The test start and end banners have been sent to the console.
+
+.. index:: test state failure
+``failure``
+  The test start banner has been sent to the console and no end banner has been
+  seen when a target restart is detected.
+
+.. index:: test state expected-fail
+``excepted-fail``
+  The test is tagged as ``expected-fail`` in the RTEMS sources for this BSP and
+  outputs the banner ``*** TEST STATE: EXPECTED_FAIL``. The test is known not
+  to pass on this BSP. The RTEMS Tester will let the test run as far as it
+  can and if the test passes it is recorded as a pass in the test results
+  otherwise it is recorded as *expected-fail*.
+
+.. index:: test state indeterminate
+``indeterminate``
+  The test is tagged as ``indeterminate`` in the RTEMS sources for this BSP and
+  outputs the banner ``*** TEST STATE: INDETERMINATE``. The test may or may not
+  pass so the result is not able to be determined. The RTEMS Tester will let
+  the test run as far as it can and record the result as indeterminate.
+
+.. index:: test state user-input
+``user-input``
+  The test is tagged as ``user-input`` in the RTEMS sources and outputs the
+  banner ``*** TEST STATE: USER_INPUT``. The RTEMS Tester will reset the target
+  if the target's configuration provides a target reset command.
+
+.. index:: test state benchmark
+``benchmark``
+  The test is tagged as ``benchmark`` in the RTEMS sources and outputs the
+  banner ``*** TEST STATE: BENCHMARK``. Benchmarks can take a while to run and
+  performance is not regression tested in RTEMS. The RTEMS Tester will reset
+  the target if the target's configuration provides a target reset command.
+
+.. index:: test state timeout
+``timeout``
+  The test start banner has been sent to the console and no end banner is seen
+  within the *timeout* period and the target has not restart. A default
+  *timeout* can be set in a target configuration, a user configuration or
+  provide on the RTEMS Tester's command line using the ``--timeout`` option.
+
+.. index:: test state invalid
+``invalid``
+  The test did not output a start banner and the RTEMS Tester has detected the
+  target has restarted. This means the executable did not load correctly, the
+  RTEMS kernel did not initialize or the RTEMS kernel configuration failed for
+  this BSP.
+
+Expected Test States
+^^^^^^^^^^^^^^^^^^^^
+
+A test's expected state is set in the RTEMS kernel's testsuite. The default for
+a tested is to ``pass``. If a test is known to fail it can have it's state set
+to ``expected-fail``. Setting tests that are known to fail to ``expected-fail``
+lets everyone know a failure is not to be countered and consider a regression.
+
+Expected test states are list in test configuration files that end with the
+file extension ``.tcfg``. The testsuite supports global test configurations in
+the ``testsuite/testdata`` directory. Global test states are applied to all
+BSPs. BSPs can provide a test configuration that applies to just that BSP.
+
+The test configuration file format is::
+
+  state: test test test
+
+where ``test test test`` is a list of tests the state applies too. The ``state`` is one
+of:
+
+``include``
+  The test list is the name of a test configuration file to include
+
+``exclude``
+  The tests listed are not build. This can happen if a BSP cannot support a
+  test. For example it does not have enough memory.
+
+``expected-fail``
+  The tests listed are set to expected fail. The test will fail on the BSP
+  being built.
+
+``user-input``
+  The tests listed require user input to run and are not supported by automatic
+  testers.
+
+``indeterminate``
+  The tests listed may pass or may not, the result is not reliable.
+
+``benchmark``
+  The tests listed are benchmarks. Benchmarks are flagged and not left to
+  run to completion because they may take too long.
+
+
+
+Test Builds
+-----------
+
+The test reports the build of RTEMS being tested. The build are:
+
+.. index:: build default
+``default``
+  The build is the default. No RTEMS configure options have been used.
+
+.. index:: build posix
+``posix``
+  The build includes the POSIX API. The RTEMS configure option
+  ``--enable-posix`` has been used. The ``cpuopts.h`` define ``RTEMS_POSIX``
+  has defined and it true.
+
+.. index:: build smp
+``smp``
+  The build is an SMP kernel. The RTEMS configure option ``--enable-smp`` has
+  been used.  The ``cpuopts.h`` define ``RTEMS_SMP`` has defined and it true.
+
+.. index:: build mp
+``mp``
+  The build is an MP kernel. The RTEMS configure option
+  ``--enable-multiprocessing`` has been used.  The ``cpuopts.h`` define
+  ``RTEMS_MULTIPROCESSING`` has defined and it true.
+
+.. index:: build paravirt
+``paravirt``
+  The build is a paravirtualization kernel. The ``cpuopts.h`` define
+  ``RTEMS_PARAVIRT`` has defined and it true.
+
+.. index:: build debug
+``debug``
+  The build includes kernel debugging support. The RTEMS configure option
+  ``--enable-debug`` has been used. The ``cpuopts.h`` define ``RTEMS_DEBUG``
+  has defined and it true.
+
+.. index:: build profiling
+``profiling``
+  The build include profiling support. The RTEMS configure option
+  ``--enable-profiling`` has been used. The ``cpuopts.h`` define
+  ``RTEMS_PROFILING`` has defined and it true.
diff --git a/user/testing/tftp.rst b/user/testing/tftp.rst
new file mode 100644
index 0000000..ee2d3c6
--- /dev/null
+++ b/user/testing/tftp.rst
@@ -0,0 +1,257 @@
+.. comment SPDX-License-Identifier: CC-BY-SA-4.0
+
+.. comment: Copyright (c) 2018 Chris Johns <chrisj@rtems.org>
+.. comment: All rights reserved.
+
+TFTP and U-Boot
+---------------
+
+.. index:: TFTP, U-Boot, Testing
+
+TFTP and U-Boot provides a simple way to test RTEMS on a network capable
+target. The RTEMS Tester starts a TFTP server for each test and the target's
+boot monitor, in this case U-Boot request a file, any file, which the TFTP
+server supplies. U-Boot loads the executable and boots it using a standard
+U-Boot script.
+
+.. _fig-tester-tftp-u-boot:
+
+.. figure:: ../../images/user/test-tftp.png
+   :width: 35%
+   :alt: RTEMS Tester using TFTP and U-Boot
+   :figclass: align-center
+
+   RTEMS Tester using TFTP and U-Boot.
+
+The :ref:`fig-tester-tftp-u-boot` figure shows the structure and control flow
+of the RTEMS Tester using TFTP and U-boot. The executables are built and the
+``rtems-test`` command is run from the top of the build directory.
+
+This test mode can only support a single test job running at once. You cannot
+add more test target hardware and run the tests in parallel.
+
+Target Hardware
+^^^^^^^^^^^^^^^
+
+The RTEMS Tester TFTP and U-Boot method of testing requires:
+
+#. A target with network interface.
+
+#. U-Boot, iPXE or similar boot loader with network driver support for your
+   target hardware and support for the TFTP protocol.
+
+#. Network power of IO switch.
+
+#. Network DHCP server.
+
+#. Console interface cable that matches your target's console UART interface.
+
+#. Telnet terminal server. See :ref:`tester-consoles`.
+
+The network power or IO switch is a device that can control power or an IO pin
+over a network connection using a script-able protocol such as Telnet or
+curl. This device can be used with the target control commands.
+
+U-Boot Set Up
+~~~~~~~~~~~~~
+
+Obtain a working image of the U-Boot boot loader for your target. We suggest
+you follow the instructions for you target.
+
+Configure U-Boot to network boot using the TFTP protocol. This is U-Boot script
+for a Zedboard::
+
+  loadaddr=0x02000000
+  uenvcmd=echo Booting RTEMS Zed from net; set autoload no; dhcp; set serverip 10.10.5.2; tftpboot zed/rtems.img; bootm; reset;
+
+The load address variable ``loadaddr`` is specific to the Zedboard and can be
+found in the various examples scripts on the internet. The script then sets
+U-Boot environment variable ``autoload`` to ``no`` causing DHCP to only request
+a DHCP lease from the DHCP server. The script sets the ``serverip`` to the host
+that will be running the RTEMS Tester then issues a TFTP request. The file name
+can be anything because the RTEMS Tester ignores it sending the executable
+image under test. Finally the script boots the download executable and if that
+fails the catch all ``reset`` resets the board and starts the boot process
+over.
+
+Test the target boots and U-Boot runs and obtains a valid DHCP lease. Manually
+connect the console's telnet port.
+
+BSP Configuration
+^^^^^^^^^^^^^^^^^
+
+The BSP's configuration file must contain the standard fields:
+
+- ``bsp``
+- ``arch``
+- ``jobs`` - Must be set to ``1``.
+- ``tester`` - Set to ``%{_rtscripts}/tftp.cfg``
+
+For example the Zedboard's configuration is::
+
+  [xilinx_zynq_zedboard]
+  bsp    = xilinx_zynq_zedboard
+  arch   = arm
+  jobs   = 1
+  tester = %{_rtscripts}/tftp.cfg
+
+The TFTP configuration supports the following field's:
+
+``bsp_tty_dev``
+  The target's tty console. For telnet this is a host and port pair written in
+  the standard networking format, for example ``serserver:12345``.
+
+``test_restarts``
+  The number of restarts before the test is considered ``invalid``.
+
+``target_reset_regex``
+  The target reset regular expression. This is a `Python regular expression
+  <https://docs.python.org/2/library/re.html#regular-expression-syntax>`_ used
+  to filter the console input. If a match is made something has happened during
+  the boot process that requires a reset. The ``target_reset_command``
+  is issued to perform the reset. This field is typically looks for boot loader
+  error messages that indicate the boot process as failed.
+
+``target_start_regex``
+  The target start regular expression. This also a Python regular expression to
+  filter the console input to detect if a target has reset. If a board crashes
+  running a test or at any point in time and reset this filter detects this as
+  happened and end the test with a suitable result.
+
+``target_on_command``
+  The target on command is a host shell command that is called before the first
+  test. This command powers on a target. Targets should be left powered off
+  when not running tests or the target may request TFTP downloads that are for
+  another target interfering with those test results. We recommend you
+  implement this command as a target off command, a pause, then a target on
+  command.
+
+``target_off_command``
+  The target off command is a host shell command that is called after the last
+  test powering off the target.
+
+``target_reset_command``
+  The target reset command is a host shell command that is called when the
+  target needs to be reset. This command can power cycle the target or toggle a
+  reset signal connected to the target. If you are power cycling a target make
+  sure you have a suitable pause to let the target completely power down.
+
+``target_pretest_command``
+  The target pretest command is a host shell comment that is called before the
+  test is run
+
+The commands in the listed fields can include parameters that are
+substituted. The parameters are:
+
+``@ARCH@``
+ The BSP architecture
+
+``@BSP@``
+ The BSP's name
+
+``@EXE@``
+  The executable name.
+
+``@FEXE@``
+ The
+. The
+  ``@ARCH`` is the
+
+substituted
+
+Some of these field are normally provided by a user's configuration. To do this
+use::
+
+  requires = bsp_tty_dev, target_on_command, target_off_command, target_reset_command
+
+The ``requires`` value requires the user provide these settings in their
+configuration file.
+
+The Zedboard's configuration file is::
+
+  [xilinx_zynq_zedboard]
+  bsp                = xilinx_zynq_zedboard
+  arch               = arm
+  jobs               = 1
+  tester             = %{_rtscripts}/tftp.cfg
+  test_restarts      = 3
+  target_reset_regex = ^No ethernet found.*|^BOOTP broadcast 6.*|^.+complete\.+ TIMEOUT.*
+  target_start_regex = ^U-Boot SPL .*
+  requires           = target_on_command, target_off_command, target_reset_command, bsp_tty_dev
+
+The ``target_start_regex`` searches for U-Boot's first console message. This
+indicate the board can restarted.
+
+The ``target_reset_regex`` checks if no ethernet interface is found. This can
+happen if U-Boot cannot detect the PHY device. It also checks if too many DHCP
+requests happen and finally a check is made for any timeouts reported by
+U-Boot.
+
+An example of a user configuration for the Zedboard is::
+
+  [xilinx_zynq_zedboard]
+  bsp_tty_dev            = selserver:12345
+  target_pretest_command = zynq-mkimg @EXE@
+  target_exe_filter      = /\.exe/.exe.img/
+  target_on_command      = power-ctl toggle-on 1 4
+  target_off_command     = power-ctl off 1
+  target_reset_command   = power-ctl toggle-on 1 3
+
+TFTP Sequences
+^^^^^^^^^^^^^^
+
+Running a large number of tests on real hardware exposes a range of issues and
+RTEMS Tester is designed to be tolerant of failures in booting or loading that
+can happen, for example a hardware design. These sequence diagrams document
+some of the sequences that can occur when errors happen.
+
+The simplest sequence is running a test. The target is powered on, the test is
+loaded and executed and a pass or fail is determined:
+
+.. _fig-tester-tftp-seq-1:
+
+.. figure:: ../../images/user/test-tftp-seq-1.png
+   :width: 90%
+   :alt: Test Pass and Fail Sequence
+   :figclass: align-center
+
+   Test Pass and Fail Sequences
+
+The target start filter triggers if a start condition is detected. This can
+happen if the board crashes or resets with no output. If this happens
+repeatedly the test result is invalid:
+
+.. _fig-tester-tftp-seq-2:
+
+.. figure:: ../../images/user/test-tftp-seq-2.png
+   :width: 80%
+   :alt: Target Start Filter Trigger
+   :figclass: align-center
+
+   Target Start Filter Trigger
+
+The reset filter triggers if an error condition is found such as the bootloader
+not being able to load the test executable. If the filter triggers the
+``target_reset_command`` is run:
+
+.. _fig-tester-tftp-seq-3:
+
+.. figure:: ../../images/user/test-tftp-seq-3.png
+   :width: 50%
+   :alt: Target Reset Filter Trigger
+   :figclass: align-center
+
+   Target Reset Filter Trigger
+
+If the RTEMS Tester does not detect a test has started it can restart the test
+by resetting the target. The reset command can toggle an IO pin connected to
+reset, request a JTAG pod issue a reset or turn the power off and on:
+
+.. _fig-tester-tftp-seq-4:
+
+.. figure:: ../../images/user/test-tftp-seq-4.png
+   :width: 60%
+   :alt: Target Timeout
+   :figclass: align-center
+
+   Target Timeout
diff --git a/user/tools/tester.rst b/user/tools/tester.rst
index 0c07415..417daa5 100644
--- a/user/tools/tester.rst
+++ b/user/tools/tester.rst
@@ -5,8 +5,8 @@
 
 .. _rtems-tester-command:
 
-RTEMS Tester and Run Commands
-=============================
+RTEMS Tester and Run
+====================
 
 .. index:: Tools, rtems-test, rtems-run