summaryrefslogtreecommitdiffstats
path: root/doc/rtems-tester.txt
blob: b6f62e2344bfc6e7039016137f7ad61075343248 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
RTEMS Tester
============
:doctype: book
:toc2:
:toclevels: 5
:icons:
:numbered:

image:images/rtemswhitebg.jpg["RTEMS",width="30%"]

Chris Johns <chrisj@rtems.org>
1.0, June 2014

RTEMS Tester
------------

The RTEMS Tester is a test framework. It includes a command line interface to
run tests on supported targets. The framework provides back-end support for
common simulators and debuggers. The board support package (BSP) configurations
for RTEMS are provided and can be used to run all the tests provided with
RTEMS. The framework is not specific to RTEMS and can be configured to run any
suitable application.

RTEMS is an embedded operating system and is cross-compiled on a range of host
machines. The executables run on the target hardware and this can vary widely
from open source simulators, commercial simulators, debuggers with simulators,
to debuggers with hardware specific pods and devices. Testing RTEMS requires
the cross-compiled test executable is transferred to the target hardware,
executed and the output returned to the host where it is analyzed to determine
the test result. The RTEMS Tester provides a framework to do this.

Running all the RTEMS tests on your target is very important. It provides you
with a traceable record your RTEMS version and its tools are working at the
level the RTEMS development team expect when releasing RTEMS. Being able to
easily run the tests and verify the results is critical in maintaining a high
standard.

The RTEMS Tester contains:

* Command line tool (+rtems-test+)
* BSP Configuration scripts
* Back-end Configuration scripts
* Back-end Python classes
* Python based framework

IMPORTANT: If you have a problem please see the <<_bugs,reporting bugs>>
           section.

This document is for users of the RTEMS Tester as well as those wanting to
develop, extend and investigate the RTEMS Tester.

License
-------

The RTEMS Tester is part of the RTEMS Tools Project. The code is released under
the OSI approved The BSD 2-Clause License. It is free to use and we encourage
this, including on operating systems other than RTEMS.

The code and command line tools must retain the same names and always reference
the RTEMS Tools Project.

Quick Start
-----------

The quick start will show you how to run the test suite for a BSP. It will
explain how to get the RTEMS Tester, set it up and run the tests for the SIS
BSP. It assumes you have a valid SPARC tool chain and have built the SIS BSP
version of RTEMS. 4.11.

Setup
~~~~~

Set up a development work space:

-------------------------------------------------------------
$ cd
$ mkdir -p development/rtems/test
$ cd development/rtems/test
-------------------------------------------------------------

First fetch the RTEMS tester from the RTEMS Tools repository::

-------------------------------------------------------------
$ git clone git://git.rtems.org/rtems-tools.git rtems-tools.git
$ cd rtems-tools.git/tester
-------------------------------------------------------------

Available BSP testers
~~~~~~~~~~~~~~~~~~~~~

You can list the available BSP testers with:

-------------------------------------------------------------
$ ./rtems-test --list-bsps
  arm920
  beagleboardxm
  jmr3904-run
  jmr3904
  mcf5235
  psim-run
  psim
  realview_pbx_a9_qemu
  sis-run
  sis
  xilinx_zynq_a9_qemu
  xilinx_zynq_a9_qemu_smp
  xilinx_zynq_zc706
  xilinx_zynq_zc706_qemu
-------------------------------------------------------------

[TIP]
=============================================================

The list is growing all the time and if your BSP is not supported we encourage
you to add it and submit the configuration back to the project.

=============================================================

Some of the BSPs may appear more than once in the list. These are aliased BSP
configurations that may use a different back-end. An example is the SPARC
Instruction Simulator (SIS) BSP. There is the 'sis' tester which uses the GDB
back-end and the 'sis-run' tester which uses the command line version of the SIS
simulator. We will show how to use +rtems-test+ command with the SIS BSP
because it is easy to build an use.

Building RTEMS Tests
~~~~~~~~~~~~~~~~~~~~

Build RTEMS with a configuration command line something similar to:

[NOTE]
=============================================================
The following assumes a Unix-type host and that the tools have been built with
a prefix of +$HOME/development/rtems/4.11+.
=============================================================

-------------------------------------------------------------
$ cd
$ export PATH=$HOME/development/rtems/4.11/bin:$PATH
$ mkdir -p development/rtems/kernel
$ cd development/rtems/kernel
$ git clone git://git.rtems.org/rtems.git rtems.git
$ cd rtems.git
$ ./bootstrap -c && ./bootstrap -p && ./bootstrap
$ cd ..
$ mkdir -p builds/sis
$ cd builds/sis
$ ../../rtems.git/configure --target=sparc-rtems4.11 \
                    --enable-tests --enable-rtemsbsp=sis
$ make <1>
-------------------------------------------------------------
<1> Add the -j option with the number of cores to run the build parallel where
it can.

Building all the tests takes time and it uses more disk so be patient. When
finished all the tests will be built and ready to run. Before running all the tests
it is a good idea to run the +hello+ test. The +hello+ test is an RTEMS version
of the classic "Hello World" example and running it shows you have a working
tool chain and build of RTEMS ready to run the tests. Using the run command:

-------------------------------------------------------------
$ sparc-rtems4.11-run <1> sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe

*** BEGIN OF TEST HELLO WORLD ***
Hello World
*** END OF TEST HELLO WORLD ***
-------------------------------------------------------------
<1> The run command is the GDB simulator without the GDB part.

Running the example using GDB:

-------------------------------------------------------------
$ sparc-rtems4.11-gdb sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
GNU gdb (GDB) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-apple-darwin13.2.0
--target=sparc-rtems4.11".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from
sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe...done.
(gdb) target sim
Connected to the simulator.
(gdb) load
(gdb) r <1>
Starting program: sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe


*** BEGIN OF TEST HELLO WORLD ***
Hello World
*** END OF TEST HELLO WORLD ***
[Inferior 1 (process 42000) exited normally]
(gdb) q
-------------------------------------------------------------
<1> To debug set break points before issuing the GDB run command.

There are currently close to 500 separate tests and you can run them all with a
single RTEMS Tester command.

TIP: Use a recent RSB version to build a SPARC tool chain. A recent patch
fixes issues with the GDB simulator and it improves the test results. It is
also recommended you use a recent RTEMS 4.11 version as the tests have been
cleaned up to improve the results.

Running the Tests
~~~~~~~~~~~~~~~~~

The +rtems-test+ command line accepts a range of options. These are discussed
later in the manual. Any command line argument without a +--+ prefix is a test
executable. You can pass more than one executable on the command line. If the
executable is a path to a directory the directories under that path are
searched for any file with a +.exe+ extension. This is the default extension
for RTEMS executables built within RTEMS.

To run the SIS tests enter the following command from the top of the SIS BSP
build tree:

-------------------------------------------------------------
$ ~/development/rtems/test/rtems-tools.git/tester/rtems-test <1> \
             --log=log_sis_run <2> \
             --rtems-bsp=sis-run <3> \
             --rtems-tools=$HOME/development/rtems/4.11 <4> \
                 sparc-rtems4.11/c/sis/testsuites <5>
RTEMS Testing - Tester, v0.2.0
[  1/500] p:0   f:0   t:0   i:0   | sparc/sis: fsbdpart01.exe
[  2/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsformat01.exe
[  5/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfswrite01.exe
[  4/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfssync01.exe
[  7/500] p:0   f:0   t:0   i:0   | sparc/sis: fsimfsgeneric01.exe
[  3/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsname01.exe
[  8/500] p:0   f:0   t:0   i:0   | sparc/sis: fsnofs01.exe
[  6/500] p:0   f:0   t:0   i:0   | sparc/sis: fsfseeko01.exe
[  9/500] p:5   f:0   t:0   i:0   | sparc/sis: fsrfsbitmap01.exe
[ 10/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fserror.exe
[ 12/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fspatheval.exe
  ................................................. <6>
[492/500] p:482 f:0   t:2   i:0   | sparc/sis: tm25.exe
[493/500] p:483 f:0   t:2   i:0   | sparc/sis: tm26.exe
[495/500] p:485 f:0   t:2   i:0   | sparc/sis: tm28.exe
[494/500] p:485 f:0   t:2   i:0   | sparc/sis: tm27.exe
[496/500] p:487 f:0   t:2   i:0   | sparc/sis: tm29.exe
[497/500] p:487 f:0   t:2   i:0   | sparc/sis: tm30.exe
[498/500] p:489 f:0   t:2   i:0   | sparc/sis: tmck.exe
[499/500] p:490 f:0   t:2   i:0   | sparc/sis: tmcontext01.exe
[500/500] p:490 f:0   t:2   i:0   | sparc/sis: tmoverhd.exe
Passed:   495 <7>
Failed:     0
Timeouts:   5
Invalid:    0
Total:    500
Average test time: 0:00:00.608249 <8>
Testing time     : 0:05:04.124696
-------------------------------------------------------------
<1> The RTEMS Tester's test command. In this example we are using an absolute
path.
<2> The +--log+ option sends the output to a log file. By default only failed
tests log the complete output.
<3> Select the SIS BSP and use GDB.
<4> Path to the RTEMS tools so GDB can be found.
<5> Path to the SIS BSP built with all tests to run. If you add subdirectories
to the path specific tests can be run.
<6> The output has been shortened so it fits nicely here.
<7> The test results. It shows passes, fails, timeouts, and invalid results. In
this run 495 tests passed and 5 tests timed out. The timeouts are probably due
to the tests not having enough execute time to complete. The default timeout
is 180 seconds and some of the interrupt tests need longer. The amount of time
depends on the performance of your host CPU running the simulations.
<8> The average time per test and the total time taken to run all the tests.

This BSP requires the +--rtems-tools+ option because the SPARC GDB is the
+sparc-rtems4.11-gdb+ command that is part of the RTEMS tools. Not every BSP
will require this option so you will need to check the specifics of the BSP
configuration to determine if it is needed.

The output you see is each test starting to run. The +rtems-test+ command by
default runs multiple tests in parallel so you will see a number start quickly
and then new tests start as others finish. The output shown here is from an
8 core processor so the first 8 are started in parallel and the status shows
the order in which they actually started, which is not 1 to 8.

The test start line shows the current status of the tests. The status reported
is when the test starts and not the result of that test. A fail, timeout or
invalid count changing means a test running before this test started failed,
not the starting test. The status here has 495 tests passed, no failures and 5
timeouts.:

-------------------------------------------------------------
[295/500]<1> p:287<2> f:0<3>  t:1<4>  i:0<5> | sparc/sis:<6> sp26.exe<7>
-------------------------------------------------------------
<1> The test number, in this case test 295 of 500 tests.
<2> Passed test count.
<3> Failed test count.
<4> Timeout test count.
<5> Invalid test count.
<6> Architecture and BSP.
<7> Executable name.

The test log records all the tests and results. The reporting mode by default
only provides the output history if a test fails, times out, or is invalid. The
time taken by each test is also recorded.

The tests must complete in a specified time or the test is marked as timed
out. The default timeout is 3 minutes and can be globally changed using the
+--timeout+ command line option. The time required to complete a test can
vary. When simulators are run in parallel the time taken depends on the
specifics of the host machine being used. A test per core is the most stable
method even though more tests can be run than available cores. If your machine
needs longer or you are using a VM you may need to lengthen the timeout.

Test Status
~~~~~~~~~~~

Tests can be marked with one of the following:

. Pass
. Fail
. Timeout
. Invalid

The RTEMS console or stdout output from the test is needed to determine the
result of the test.

.Pass
A test passes if the start and end markers are seen in the test output. The
start marker is "+\*\*\*+" and the end mark is "+\*** END OF TEST+". All tests in the
RTEMS test suite have these markers.

.Fail
A test fails if the start marker is seen and there is no end marker.

.Timeout
If the test does not complete within the timeout setting the test is marked as
having timed out.

.Invalid
If no start marker is seen the test is marked as invalid. If you are testing on
real target hardware things can sometimes go wrong and the target may not
initialize or respond to the debugger in an expected way.

Reporting
~~~~~~~~~

The report written to the log has the following modes:

. All (+all+)
. Failures (+failures+)
. None (+none+)

The mode is controlled using the command line option +--report-mode+ using the
values listed above.

.All
The output of all tests is written to the log.

.Failures
The output of the all tests that do not pass is written to the log.

.None
No output is written to the log.

The output is tagged so you can determine where it comes from. The following is
the complete output for the In Memory File System test +imfs_fslink.exe+
running on a Coldfire MCF5235 using GDB and a BDM pod:

-------------------------------------------------------------
[ 11/472] p:9   f:0   t:0   i:1   | m68k/mcf5235: imfs_fslink.exe
> gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe <1>
> Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe...
> done. <2>
> target remote | m68k-bdm-gdbserver pipe 003-005
> Remote debugging using | m68k-bdm-gdbserver pipe 003-005
> m68k-bdm: debug module version 0
> m68k-bdm: detected MCF5235
> m68k-bdm: architecture CF5235 connected to 003-005
> m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282)
> Process 003-005 created; pid = 0
> 0x00006200 in ?? ()
> thb *0xffe254c0
> Hardware assisted breakpoint 1 at 0xffe254c0
> continue
> Continuing.
] <3>
]
] External Reset
]
] ColdFire MCF5235 on the BCC
] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28)
] Copyright 1995-2004 Freescale Semiconductor, Inc.  All Rights Reserved.
]
] Enter 'help' for help.
]
> Temporary breakpoint
> 1, 0xffe254c0 in ?? ()
> load
> Loading section .text, size 0x147e0 lma 0x40000
> Loading section .data, size 0x5d0 lma 0x547e0
> Start address 0x40414, load size 85424
> Transfer rate: 10 KB/sec, 1898 bytes/write.
> b bsp_reset
> Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14.
> continue
> Continuing.
] dBUG>
]
] *** FILE SYSTEM TEST ( IMFS ) *** <4>
] Initializing filesystem IMFS
]
]
] *** LINK TEST ***
] link creates hardlinks
] test if the stat is the same
] chmod and chown
] unlink then stat the file
] *** END OF LINK TEST ***
]
]
] Shutting down filesystem IMFS
] *** END OF FILE SYSTEM TEST ( IMFS ) *** <5>
> Breakpoint
> 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14
> 14    {
Result: passed     Time: 0:00:10.045447 <6>
-------------------------------------------------------------
<1> GDB command line (Note: paths with \'....' have been shortened)
<2> Lines starting with +>+ are from GDB's console.
<3> Line starting with +]+ are from the target's console.
<4> The start marker.
<5> The end marker.
<6> The result with the test time.

Running Tests in Parallel
-------------------------

The RTEMS Tester supports parallel execution of tests by default. This only
makes sense if the test back-end can run in parallel without resulting in
resource contention. Simulators are an example of back-ends that can run in
parallel. A hardware debug tool like a BDM or JTAG pod can manage only a
single test at once so the tests need to be run one at a time.

The test framework manages the test jobs and orders the output in the report
log in test order. Output is held for completed tests until the next test to be
reported has finished.

Command Line Help
~~~~~~~~~~~~~~~~~

The +rtems-test+ command line accepts a range of options. You can review the
available option by the +--help+ option:

-------------------------------------------------------------
RTEMS Tools Project (c) 2012-2014 Chris Johns
Options and arguments:
--always-clean               : Always clean the build tree, even with an error
--debug-trace                : Debug trace based on specific flags
--dry-run                    : Do everything but actually run the build
--force                      : Force the build to proceed
--jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs.
--keep-going                 : Do not stop on an error.
--list-bsps                  : List the supported BSPs
--log file                   : Log file where all build output is written to
--macros file[,file]         : Macro format files to load after the defaults
--no-clean                   : Do not clean up the build tree
--quiet                      : Quiet output (not used)
--report-mode                : Reporting modes, failures (default),all,none
--rtems-bsp                  : The RTEMS BSP to run the test on
--rtems-tools                : The path to the RTEMS tools
--target                     : Set the target triplet
--timeout                    : Set the test timeout in seconds (default 180 seconds)
--trace                      : Trace the execution
--warn-all                   : Generate warnings
-------------------------------------------------------------

Development
------------

The RTEMS Tester framework and command line tool is under active
development. These are changing, being fixed, broken and generally improved. If
you want to help please see the Wiki page for open items.



History
-------

The RTEMS Tester is based on a refactored base of Python code used in the RTEMS
Source Builder. This code provided a working tested base that has been extended
and expanded to meet the requirements for the RTEMS Tester. The tester uses the
specifics found in the various scripts and configurations in the
rtems-testing.git repo that has been accumulated over many years. The shell
script implementation is restricted in what it can do and, per BSP script, is a
maintenance burden. For example the command lines and options vary between each
script.