RTEMS BSD USB and TCP/IP Developers Guide
=========================================
Joel Sherrill <joel.sherrill@oarcorp.com>
:Author Initials: JRS
:toc:
:icons:
:numbered:
:website: http://www.rtems.org/
RTEMS uses FreeBSD as the source of its TCP/IP and USB stacks.
This is a developers guide which captures information on the
process of merging code from FreeBSD, building this library,
RTEMS specific support files, and general guidelines on what
modifications to the FreeBSD source are permitted.
Goals of this effort are:
* Update TCP/IP and provide USB in RTEMS
* Ease updating to future FreeBSD versions
* Ease tracking changes in FreeBSD code
* Minimize manual changes in FreeBSD code
* Define stable kernel/device driver API which is implemented
by both RTEMS and FreeBSD. This is the foundation of the port.
We will work to push our changes upstream to the FreeBSD Project
and minimize changes required at each update point.
**************************************************************
This is a work in progress and is very likely to be incomplete.
Please help by adding to it.
**************************************************************
== Source Code Version Information
* FreeBSD 8.2 SVN r222496
* RTEMS 4.11
- BSP must have support for all new BSD sys sections
- It is preferable if the BSP uses linkcmds.base.
- BSP must be from an architecture with Programmable Interrupt Controller
interrupt model.
The FreeBSD 8.2 SVN checkout will generally be referred to as the
FreeBSD source in this document. An archive of the FreeBSD 8.2 SVN
archive used is located at
http://www.rtems.org/ftp/pub/rtems/people/joel/freebsd/
The SVN checkout command is this
svn co http://svn.freebsd.org/base/releng/8.2/sys/ -r222496 freebsd-8.2
== Issues and To Do
* Sebastian Huber: mentioned some simple test code which would verify
that the BSD code/and or USB stack was initialized. This has been
sent to Joel Sherrill and is pending merger.
* Sebastian Huber and Joel Sherrill discussed the need for a a basic USB
functionality test that is known to work on qemu pc.
* Adapt generic IRQ PIC interface code to Simple Vectored Interrupt Model
so that those architectures can use new TCP/IP and USB code.
* in_cksum implementations for architectures not supported by FreeBSD.
This will require figuring out where to put implementations that do
not originate from FreeBSD and are populated via the script.
* FreeBSD generic in_cksum implementation is missing in_cksum_split so
currently cannot be used.
* How does one initialize the TCP/IP stack?
* linker section issues: I have undefined symbols for
`_bsd__start_set_sysinit_set` and `_bsd__stop_set_sysinit_set`.
Is this the only type of new section magic? What about the old sysctl_set?
I added this to my linkcmds.
[listing]
----
/* sysinit section? */
. = ALIGN (16);
_bsd__start_set_sysinit_set = .;
*(set_sys_init_*);
_bsd__stop_set_sysinit_set = .;
----
* Why is the interrupt server used? The BSD interrupt handlers can block on
synchronization primitives like mutexes. This is in contrast to RTEMS
interrupt service routines. The BSPs using the generic interrupt support must
implement the `bsp_interrupt_vector_enable()` and
`bsp_interrupt_vector_disable()` routines. They normally enable/disable a
particular interrupt source at the interrupt controller. This can be used to
implement the interrupt server. The interrupt server is a task that wakes-up
in case an associated interrupt happens. The interrupt source is disabled in
a generic interrupt handler that wakes-up the interrupt server task. Once the
postponed interrupt processing is performed in the interrupt server the
interrupt source is enabled again.
* Convert all BSP linkcmds to use a linkcmds.base so the sections are
easier to insert.
* rtems-bsd-init-with-irq.c:
rtems_bsd_initialize_with_interrupt_server() has reference to
rtems_interrupt_server_initialize() and this method is unimplemented
- XXX BSP implements pieces
- BSPs using this software stack must support it apparently.
- What about Simple Vectored architectures?
* maxproc variable referenced by rtems-bsd-resource.c. What should it
be set to?
* ngroups_max variable referenced by rtems-bsd-prot.c. - What should
it be set to?
* NIC Device Drivers
- Only common NIC drivers have been included in the initial set. These do not
include any system on chip or ISA drivers.
- The ISA drivers require more BSD infrastructure to be addressed. This was
outside the scope of the initial porting effort.
== FreeBSD Source
You should be able to rely on FreebSD manual pages and documentation
for details on the code itself.
=== Automatically Generated FreeBSD Files
The FreeBSD source tarball includes a file named Makefile.rtems which
has stanzas to automatically generate some files using awk. For details
on this, see http://www.freebsd.org/cgi/man.cgi?query=kobj&apropos=0&sektion=0&manpath=FreeBSD+9.0-RELEASE&arch=default&format=html
XXX This needs more detail.
=== Rules for Modifying FreeBSD Source
* Only add lines. Subtract code by added "ifndef __rtems__". This makes
merging easier in the future.
== libbsd Source
=== What is in git
The git source is a self-contained kit with FreeBSD and RTEMS components
pre-merged. The Makefile in this kit is automatically generated.
Any changes to sources in the freebsd or contrib directories will need to
be merged upstream into our master FreeBSD svn checkout.
The FreeBSD sources managed in RTEMS libbsd git repository (e.g. contrib
and freebsd directories) contain the "managed" version of the
FreeBSD source. The FreeBSD SVN source is the "master" version. The
freebsd-to-rtems.py script is used to transfer files between the two
trees. In general terms, if you have modified FreeBSD in the RTEMS libbsd
tree, you want to run the script in "revert" or "reverse" mode to move
the source back so you can run an "svn diff" against the upstream FreeBSD
source. If you want to transfer source from the FreeBSD SVN checkout to
the RTEMS libbsd tree, then you want to run the script in "forward" or
default mode.
=== Building RTEMS libbsd source
You need to configure RTEMS for the desired BSP and install it. The
following is the script used to build the powerpc/psim BSP for our
internal testing purposes:
[listing]
----
#! /bin/sh
cd ${HOME}/newbsd
rm -rf b-psim
mkdir b-psim
cd b-psim
../git/rtems/configure --target=powerpc-rtems4.11 \
--enable-rtemsbsp=psim --disable-networking \
--enable-tests=samples \
--prefix=${HOME}/newbsd/bsp-install >c.log 2>&1 && \
make >b.log 2>&1 && \
make install >i.log 2>&1
echo $?
----
Then edit the file config.inc to set RTEMS_MAKEFILE_PATH appropriately
to indicate the ${prefix}/${target}/${BSP}. Continuing on the above,
the config.inc used to match the above is:
[listing]
----
RTEMS_MAKEFILE_PATH = ${HOME}/newbsd/bsp-install/powerpc-rtems4.11/psim/
INSTALL_BASE = ${HOME}/newbsd/install
----
The above installs the RTEMS libbsd kit into a separate place from
RTEMS and the BSP. The RTEMS libbsd tests are built against an installed
image of the RTEMS libbsd. By keeping it in a separate installation point
from RTEMS itself, this makes it easier to remove a libbsd installation
and have a clean test point.
[listing]
----
make
make install
make -C testsuite
----
At this point, we expect multiple linker errors. That is what we are
currently working on.
=== Organization
The top level directory contains a few directories and files. The following
are important to understand:
* freebsd-to-rtems.py - script to convert to and free FreeBSD and RTEMS trees
* Makefile - automatically generated
* contrib/ - from FreeBSD by script.
* freebsd/ - from FreeBSD by script.
* rtemsbsd/ - RTEMS specific implementations of FreeBSD kernel support routines.
* testsuite/ - RTEMS specific tests
* libbsd.txt - Documentation in Asciidoc
== Moving Code Between FreeBSD SVN and RTEMS libbsd
The script freebsd-to-rtems.py is used to copy code from FreeBSD to the
RTEMS libbsd tree and to reverse this process. This script attempts to
automate this process as much as possible and performs some transformations
on the FreeBSD code. Its command line arguments are shown below:
[listing]
----
freebsd-to-rtems.py [args]
-?|-h|--help print this and exit
-d|--dry-run run program but no modifications
-D|--diff provide diff of files between trees
-e|--early-exit evaluate arguments, print results, and exit
-m|--makefile just generate Makefile
-R|--reverse default FreeBSD -> RTEMS, reverse that
-r|--rtems RTEMS directory
-f|--freebsd FreeBSD directory
-v|--verbose enable verbose output mode
----
In its default mode of operation, freebsd-to-rtems.py is used to copy code
from FreeBSD to the RTEMS libbsd tree and perform transformations. In forward
mode, the script may be requested to just generate the Makefile.
In "reverse mode", this script undoes those transformations and copies
the source code back to the FreeBSD SVN tree. This allows us to do
'svn diff', evaluate changes made by the RTEMS Project, and report changes
back to FreeBSD upstream.
In either mode, the script may be asked to perform a dry-run or be verbose.
Also, in either mode, the script is also smart enough to avoid copying over
files which have not changed. This means that the timestamps of files are
not changed unless the contents change. The script will also report the
number of files which changed. In verbose mode, the script will print
the name of the files which are changed.
The following is an example forward run with no changes.
[listing]
----
$ ~/newbsd/git/libbsd-8.2/freebsd-to-rtems.py \
-r /home/joel/newbsd/git/libbsd-8.2 \
-f /home/joel/newbsd/libbsd/freebsd-8.2 -v
Verbose: yes
Dry Run: no
Only Generate Makefile: no
RTEMS Directory: /home/joel/newbsd/git/libbsd-8.2
FreeBSD Directory: /home/joel/newbsd/libbsd/freebsd-8.2
Direction: forward
Generating into /home/joel/newbsd/git/libbsd-8.2
0 files were changed.
----
The script may also be used to generate a diff in either forward or reverse
direction.
== Initialization of RTEMS libbsd
The initialization of the RTEMS libbsd is based on the FreeBSD SYSINIT(9)
infrastructure. The key to initializing a system is to ensure that the desired
device drivers are explicitly pulled into the linked application. This plus
linking against the libbsd library will pull in the necessary FreeBSD
infrastructure.
The FreeBSD kernel is not a library like the RTEMS kernel. It is a bunch of
object files linked together. If we have a library, then creating the
executable is simple. We begin with a start symbol and recursively resolve all
references. With a bunch of object files linked together we need a different
mechanism. Most object files don't know each other. Lets say we have a driver
module. The rest of the system has no references to this driver module. The
driver module needs a way to tell the rest of the system: Hey, kernel I am
here, please use my services!
This registration of independent components is performed by SYSINIT(9) and
specializations:
http://www.freebsd.org/cgi/man.cgi?query=SYSINIT
The SYSINIT(9) uses some global data structures that are placed in a certain
section. In the linker command file we need this:
[listing]
----
.robsdsets : {
_bsd__start_set_modmetadata_set = .;
*(_bsd_set_modmetadata_set);
_bsd__stop_set_modmetadata_set = .;
_bsd__start_set_sysctl_set = .;
*(_bsd_set_sysctl_set);
_bsd__stop_set_sysctl_set = .;
} > REGION_RODATA AT > REGION_RODATA_LOAD
.rwbsdsets : {
_bsd__start_set_sysinit_set = .;
*(_bsd_set_sysinit_set);
_bsd__stop_set_sysinit_set = .;
} > REGION_DATA AT > REGION_DATA_LOAD
----
Here you can see, that these global data structures are collected into
continuous memory areas. This memory area can be identified by start and stop
symbols. This constructs a table of uniform items.
The low level FreeBSD code calls at some time during the initialization the
mi_startup() function (machine independent startup). This function will sort
the SYSINIT(9) set and call handler functions which perform further
initialization. The last step is the scheduler invocation.
The SYSINIT(9) routines are run in mi_startup() which is called by
rtems_bsd_initialize().
This is also explained in "The Design and Implementation of the FreeBSD
Operating System" section 14.3 "Kernel Initialization".
In RTEMS we have a library and not a bunch of object files. Thus we need a way
to pull-in the desired services out of the libbsd. Here the
"rtems-bsd-sysinit.h" comes into play. The SYSINIT(9) macros have been
modified and extended for RTEMS in "sys/kernel.h":
[listing]
----
#ifndef __rtems__
#define C_SYSINIT(uniquifier, subsystem, order, func, ident) \
static struct sysinit uniquifier ## _sys_init = { \
subsystem, \
order, \
func, \
(ident) \
}; \
DATA_SET(sysinit_set,uniquifier ## _sys_init)
#else /* __rtems__ */
#define SYSINIT_ENTRY_NAME(uniquifier) \
_bsd_ ## uniquifier ## _sys_init
#define SYSINIT_REFERENCE_NAME(uniquifier) \
_bsd_ ## uniquifier ## _sys_init_ref
#define C_SYSINIT(uniquifier, subsystem, order, func, ident) \
struct sysinit SYSINIT_ENTRY_NAME(uniquifier) = { \
subsystem, \
order, \
func, \
(ident) \
}; \
DATA_SET(sysinit_set,SYSINIT_ENTRY_NAME(uniquifier))
#define SYSINIT_REFERENCE(uniquifier) \
extern struct sysinit SYSINIT_ENTRY_NAME(uniquifier); \
static struct sysinit const * const \
SYSINIT_REFERENCE_NAME(uniquifier) __used \
= &SYSINIT_ENTRY_NAME(uniquifier)
#define SYSINIT_MODULE_REFERENCE(mod) \
SYSINIT_REFERENCE(mod ## module)
#define SYSINIT_DRIVER_REFERENCE(driver, bus) \
SYSINIT_MODULE_REFERENCE(driver ## _ ## bus)
#endif /* __rtems__ */
----
Here you see that the SYSINIT(9) entries are no longer static. The
*_REFERENCE() macros will create references to the corresponding modules which
are later resolved by the linker. The application has to provide an object
file with references to all required FreeBSD modules.
The FreeBSD device model is quite elaborated (with follow-ups):
http://www.freebsd.org/cgi/man.cgi?query=driver
The devices form a tree with the Nexus device at a high-level. This Nexus
device is architecture specific in FreeBSD. In RTEMS we have our own Nexus
device, see "rtems-bsd-nexus.c". It uses a table to add child devices:
[listing]
----
const char *const _bsd_nexus_devices [] = {
#ifdef NEED_USB_OHCI
"ohci",
#endif
#ifdef NEED_USB_EHCI
"ehci",
#endif
#ifdef NEED_SDHC
"sdhci",
#endif
NULL
};
----
This table must be provided by the application.
=== SYSCTL_NODE Example
During development, we had an undefined reference to
_bsd_sysctl__net_children that we had trouble tracking down. Thanks to
Chris Johns, we located it. He explained how to read SYSCTL_NODE
definitions. This line from freebsd/netinet/in_proto.c is attempting
to add the "inet" node to the parent node "_net".
[listing]
----
SYSCTL_NODE(_net, PF_INET, inet, CTLFLAG_RW, 0,
"Internet Family");
----
Our problem was that we could not find where _bsd_sysctl__net_children
was defined. Chris suggested that when in doubt compile with -save-temps
and look at the preprocessed .i files. But he did not need that. He
explained that this the symbol name _bsd_sysctl__net_children was
automatically generated by a SYSCTL_NODE as follows:
* _bsd_ - added by RTEMS modifications to SYSCTL_NODE macro
* sysctl_ - boilerplace added by SYSCTL_NODE macro
* "" - empty string for parent node
* net - name of SYSCTL_NODE
* children - added by SYSCTL macros
This was all generated by a support macro declaring the node as this:
[listing]
----
struct sysctl_oid_list SYSCTL_NODE_CHILDREN(parent, name);
----
Given this information, we located this SYSCTL_NODE declaration in
kern/kern_mib.c
[listing]
----
SYSCTL_NODE(, CTL_KERN, kern, CTLFLAG_RW, 0,
"High kernel, proc, limits &c");
----
== Core FreeBSD APIs and RTEMS Replacements ==
=== SX(9) (Shared/exclusive locks) ===
http://www.freebsd.org/cgi/man.cgi?query=sx
Binary semaphores (this neglects the ability to allow shared access).
=== MUTEX(9) (Mutual exclusion) ===
http://www.freebsd.org/cgi/man.cgi?query=mutex
Binary semaphores (not recursive mutexes are not supported this way).
=== RWLOCK(9) (Reader/writer lock) ===
http://www.freebsd.org/cgi/man.cgi?query=rwlock
POSIX r/w lock.
=== RMLOCK(9) (Reader/writer lock optimized for mostly read access patterns) ===
Note: This object was implemented as a wrapper for RWLOCK in the rm_lock header file.
http://www.freebsd.org/cgi/man.cgi?query=rmlock
POSIX r/w lock.
=== CONDVAR(9) (Condition variables) ===
http://www.freebsd.org/cgi/man.cgi?query=condvar
POSIX condition variables with modifications (hack).
=== CALLOUT(9) (Timer functions) ===
http://www.freebsd.org/cgi/man.cgi?query=callout
Timer server.
=== TASKQUEUE(9) (Asynchronous task execution) ===
http://www.freebsd.org/cgi/man.cgi?query=taskqueue
TBD.
=== KTHREAD(9), KPROC(9) (Tasks) ===
http://www.freebsd.org/cgi/man.cgi?query=kthread
http://www.freebsd.org/cgi/man.cgi?query=kproc
Tasks.
=== ZONE(9) (Zone allocator) ===
http://www.freebsd.org/cgi/man.cgi?query=zone
TBD.
=== devfs (Device file system) ===
Dummy, IMFS or new implementation (currently dummy).
=== psignal (Signals) ===
TBD. Seems to be not needed.
=== poll, select ===
TBD. Seems to be not needed.
=== RMAN(9) (Resource management) ===
http://www.freebsd.org/cgi/man.cgi?query=rman
TBD. Seems to be not needed.
=== DEVCLASS(9), DEVICE(9), DRIVER(9), MAKE_DEV(9) (Device management) ===
http://www.freebsd.org/cgi/man.cgi?query=devclass
http://www.freebsd.org/cgi/man.cgi?query=device
http://www.freebsd.org/cgi/man.cgi?query=driver
http://www.freebsd.org/cgi/man.cgi?query=make_dev
Use FreeBSD implementation as far as possible. FreeBSD has a nice API for
dynamic device handling. It may be interesting for RTEMS to use this API
internally in the future.
=== BUS_SPACE(9), BUS_DMA(9) (Bus and DMA access) ===
http://www.freebsd.org/cgi/man.cgi?query=bus_space
http://www.freebsd.org/cgi/man.cgi?query=bus_dma
Likely BSP dependent. A default implementation for memory mapped linear access
is easy to provide. The current heap implementation supports all properties
demanded by bus_dma (including the boundary constraint).
== RTEMS Replacements by File Description and Status ==
Note: Files with a status of USB are used by the USB test and have at least
been partially tested. If they contain both USB and Nic, then they are used
by both and MAY contain methods that have not been tested yet. Files that
are only used by the Nic test are the most suspect.
rtems-libbsd File: rtems-bsd-assert.c
FreeBSD File: rtems-bsd-config.h redefines BSD_ASSERT.
Description: This file contains the support method rtems_bsd_assert_func().
Status: USB, Nic
rtems-libbsd File: rtems-bsd-autoconf.c
FreeBSD File: FreeBSD has BSP specific autoconf.c
Description: This file contains configuration methods that are used to setup the system.
Status: USB
rtems-libbsd File: rtems-bsd-bus-dma.c
FreeBSD File: FreeBSD has BSP specific busdma_machdep.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-bus-dma-mbuf.c
FreeBSD File: FreeBSD has BSP specific busdma_machdep.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-callout.c
FreeBSD File: kern/kern_timeout.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-cam.c
FreeBSD File: cam/cam_sim.c
Description:
Status: USB
rtems-libbsd File: rtems-bsd-condvar.c
FreeBSD File: kern/kern_condvar.c
Description:
Status: USB
rtems-libbsd File: rtems-bsd-copyinout.c
FreeBSD File: bsp specific copyinout.c )
Description: Note: The FreeBSD file is split with some methods being in rtems-bsd-support
Status: Nic
rtems-libbsd File: rtems-bsd-delay.c
FreeBSD File: bsp specific file with multiple names
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-descrip.c
FreeBSD File: kern/kern_descrip.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-generic.c
FreeBSD File: kern/sys_generic.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-init.c
FreeBSD File: N/A
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-init-with-irq.c
FreeBSD File: N/A
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-jail.c
FreeBSD File: kern/kern_jail.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-lock.c
FreeBSD File: kern/subr_lock.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-log.c
FreeBSD File: kern/subr_prf.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-malloc.c
FreeBSD File: kern/kern_malloc.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-mutex.c
FreeBSD File: kern/kern_mutex.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-newproc.c
FreeBSD File: N/A
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-nexus.c
FreeBSD File: bsp specific nexus.c
Description:
Status: USB
rtems-libbsd File: rtems-bsd-panic.c
FreeBSD File: boot/common/panic.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-rwlock.c
FreeBSD File: kern_rwlock.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-shell.c
FreeBSD File: N/A
Description:
Status: USB
rtems-libbsd File: rtems-bsd-signal.c
FreeBSD File: kern/kern_sig.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-smp.c
FreeBSD File: N/A
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-support.c
FreeBSD File: bsp specific copyinout.c
Description: Note: the FreeBSD file is split with some methods being in rtems-bsd-copyinout.
Status: USB, Nic
rtems-libbsd File: rtems-bsd-sx.c
FreeBSD File: kern/kern_sx.c
Description: Status: USB, Nic
rtems-libbsd File: rtems-bsd-synch.c
FreeBSD File: kern/kern_synch.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-syscalls.c
FreeBSD File: User API for kern/uipc_syscalls.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-sysctlbyname.c
FreeBSD File: User API for sysctlbyname(3)
Description:
Status:
rtems-libbsd File: rtems-bsd-sysctl.c
FreeBSD File: User API for sysctl(8)
Description:
Status:
rtems-libbsd File: rtems-bsd-sysctlnametomib.c
FreeBSD File: User API for sysctlnametomib
Description:
Status:
rtems-libbsd File: rtems-bsd-taskqueue.c
FreeBSD File: kern/subr_taskqueue.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-thread.c
FreeBSD File: kern/kern_kthread.c
Description:
Status: USB, Nic
rtems-libbsd File: rtems-bsd-timeout.c
FreeBSD File: kern/kern_timeout.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-timesupport.c
FreeBSD File: kern/kern_clock.c
Description:
Status: Nic
rtems-libbsd File: rtems-bsd-vm_glue.c
FreeBSD File: vm/vm_glue.c
Description:
Status: USB, Nic
