path: root/gcc/RTEMS_GCC_Testing.txt

// Build with the following command:
//     asciidoc -a data-uri -a icons -a toc -a max-width=55em \
//        RTEMS_GCC_Testing.txt

Testing GNU Tools on RTEMS Targets
==================================
Joel Sherrill <joel.sherrill@oarcorp.com>
v0.1, 2013-04


This document provides an overview of testing the GNU Development
Tools on RTEMS targets.

:numbered:

Introduction
------------
This document describes the process and requirements for testing
the GNU Development Tools including binutils, gcc, newlib, and gdb
on RTEMS targets.

Currently only testing on simulators is supported.

Build Order
----------
This section describes the build order for testing purposes.
It does not detail source setup including patching, symbolic
linking, or specific build isntructions, etc. It is assumed
that the install point (e.g. '--prefix') is a temporary location
that does not conflict with any other existing installations or
the host operating system tools.

*NOTE*: The order in this section is the same as that in the
'rtems-testing/gcc/do_one' script.

In some cases, particularly with secondary languages, it may be necessary to
build and install a native compile from the same source code you are building
cross. If this is needed, then the first step is to:

- Install native GNU Compiler Collection with support for *ALL* languages
you plan to build a cross-compiler for.

*NOTE*: These scripts assume that the native and cross tools are installed
into the same directory. This simplifies having unified native and
cross-toolsets for testing purposes.

With the native compiler installed and it at the head of your $PATH,
then build a cross toolset targetting a specific RTEMS target.

- Install GNU binutils
* can run tests on binutils without RTEMS
- Install GNU Debugger (includes simulator when available)
* can run tests on binutils without RTEMS
- Install GNU Compiler Collection with C/C++ support
* can *NOT* run tests on GCC without RTEMS
* includes newlib

Any failures in this part of the build are considered fatal for the
target. Without a functioning C compiler, assembler, etc. there is
not enough to build RTEMS, run gcc tests, or build the secondary
languages.

*NOTE*: There is no mailing list to submit 'binutils' or 'gdb' test results
to. They could be sent to the RTEMS Test Results mailing list but 
'rtems-testing/gcc/do_one' does not currently do this.

At this point you have enough installed to build RTEMS. The RTEMS header
files must be installed so programming languages with more complex
run-tiems like Ada and Go can be compiled. Both require some header files
that are not provided by the C Library.

- Install RTEMS built multilib.
* Build with 'make -k' since some BSPs you do not need for testing may
fail. Please report these failures.];
- Install RTEMS built for the BSP you will be testing on.

At this point, you have the infrastructure required to:

- Build and execute the GCC C/C++ tests
- Build cross compilers for each secondary language and run their tests.
Secondary languages with RTEMS support include:
* Ada
* Java (GCJ)
* Go
* FORTRAN
* Objective-C (no threading yet)

Test results for C/C++ and each secondary language are mailed to the GCC
Test Results mailing list and to the RTEMS Tools Test Results mailing list.
This is done by the 'rtems-testing/gcc/do_one' script if invoked with
the correct arguments.

The scripts in this directory build each secondary language separately.
This is based on the assumption that one secondary language may fail
but another may build. Thus we may end up with FORTRAN test results
even when Ada doesn't build.

*NOTE*: There is currently no support for checking the current test run
against a previous run for regressions.

*NOTE*: Some targets - in particular the h8300-rtems* have an excessively
high number of failures due to multilib issues and tests not fitting into the
simulator memory. Theses failures result in a test report which is too large
to be accepted by the RTEMS and GCC Test Results mailing lists.

Simulators Supported for GCC Testing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
GCC testing requires that some tess be executed on the "target". For
RTEMS GCC testing, the target is always a simulator.

Although any simulator or target hardware could be used, only a subset
of simulators supported in 'rtems-testing/sim-scripts' are supported
for GCC testing currently. The focus is on executing tests using
free and open source target simulators so any RTEMS Community Member
can duplicate these results.

The following table lists each RTEMS target and the BSP supported
for GCC testing. In addition, this table lists the simulator used.

- 'not executed' indicates that the target is built using a special
configuration which does not run the tests.
- 'not supported' indicates that there is no current support for
this target

.Targets and BSPs
[width="60%",options="header",align="center"]
|==================================================
| Target             | BSP        | Simulator
| arm-rtems*         | edb7312    | Skyeye
| avr-rtems*         | N/A        | not supported
| bfin-rtems*        | eZKit533   | Skyeye
| h8300-rtems*       | N/A        | not supported
| i386-rtems*        | pc386      | qemu
| lm32-rtems*        | lm32_evr   | gdb
| m32c-rtems*        | m32csim    | gdb
| m32r-rtems*        | N/A        | not supported
| m68k-rtems*        | uC5282     | Qemu
| microblaze-rtems*  | nosim      | not executed
| mips-rtems*        | jmr3904    | gdb
| moxie-rtems*       | N/A        | not supported
| powerpc-rtems*     | psim       | gdb
| powerpc-rtems*     | qemuppc    | qemu
| nios2-rtems*       | N/A        | not supported
| sh-rtems*          | simsh1     | gdb
| sparc-rtems*       | sis        | gdb
| sparc64-rtems*     | niagara    | not executed
| v850-rtems*        | v850*sim*  | gdb
|==================================================

Support for other simulators could be added.

DejaGNU Target Specific Support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The GNU Tools use the
link:http://www.gnu.org/software/dejagnu/[DejaGNU]
testing framework. DejaGNU includes the concept of baseboard which is
how the test execution environment is described to the framework.
'rtems-testing/dejagnu' includes the baseboard files which describe
the target simulators listed in the previous section.

In general, these baseboard files use the 'rtems-testing/sim-scripts'
simulator invocation wrapper scripts in interactive mode. DejaGNU
handles test case timeout.

DejaGNU can also be used to run test cases on real target hardware.
The RTEMS Project does not currently do this.

rundeja Script
~~~~~~~~~~~~~~
The 'rundeja' script is invoked as follows:

.rundeja Invocation
----
  rundeja BSP TESTSUITE
----

Where *BSP* is an RTEMS BSP supported by a DejaGNU baseboard configuration
file.

And *TESTSUITE* is one of the following:
- gcc
- objc
- gccgo
- libgo
- java
- fortran


*NOTE*: Testing of Ada is performed by the
'rtems-testing/gcc/testsuite/ada/acats/run_all_rtems.sh' script described
in the next section.

runall_rtems.sh Ada Testing Script
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The 'run_all_rtems.sh' script is invoked as follows:

.run_all_rtems.sh Invocation
----
  run_all_rtems.sh TOOL_INSTALL BSP_INSTALL TARGET BSP
----

Where *TOOL_INSTALL* is the installation directory of the cross-development
toolset. It is the same as the '--prefix' argument provided when configuring
each tool.

Where *BSP_INSTALL* is the installation directory of the RTEMS BSP to
be used in this test run.  It is the same as the '--prefix' argument
provided when configuring RTEMS.

Where *TARGET* is a GNU style target such as sparc-rtems4.11.

Where *BSP* is an RTEMS BSP supported by 'rtems-testing/sim_scripts'


Native GCC Version Selection
----------------------------
For at least Ada, it is recommended to use a native compiler generated
from the same source. There are technically ranges of versions which
are supposed to work at any particular time but using a precise version
match is more reliable.

This advice likely applies to other secondary languages but Ada is
known to be sensitive.