From 60ed99d2b0c01f46c3458f45c0d0cd9334b6c497 Mon Sep 17 00:00:00 2001 From: Sebastian Huber Date: Fri, 11 Jan 2019 10:20:08 +0100 Subject: user: Add RSB content as a chapter Remove the separate RSB manual. --- user/rsb/configuration.rst | 1514 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1514 insertions(+) create mode 100644 user/rsb/configuration.rst (limited to 'user/rsb/configuration.rst') diff --git a/user/rsb/configuration.rst b/user/rsb/configuration.rst new file mode 100644 index 0000000..141fd3b --- /dev/null +++ b/user/rsb/configuration.rst @@ -0,0 +1,1514 @@ +.. SPDX-License-Identifier: CC-BY-SA-4.0 + +.. Copyright (C) 2012, 2016 Chris Johns + +.. _Configuration: + +Configuration +============= + +The RTEMS Source Builder has two types of configuration data: + +- Build Sets + +- Package Build Configurations + +By default these files can be located in two separate directories and +searched. The first directory is ``config`` in your current working directory +(``_topdir``) and the second is ``config`` located in the base directory of the +RTEMS Source Builder command you run (``_sbdir``). The RTEMS directory +``rtems``` located at the top of the RTEMS Source Builder source code is an +example of a specific build configuration directory. You can create custom or +private build configurations and if you run the RTEMS Source Builder command +from that directory your configurations will be used. + +The configuration search path is a macro variable and is reference as +``%{_configdir}``. It's default is defined as:: + + _configdir : dir optional<2> %{_topdir}/config:%{_sbdir}/config <1> + +.. topic:: Items: + + 1. The ``_topdir`` is the directory you run the command from and ``_sbdir`` + is the location of the RTEMS Source Builder command. + + 2. A macro definition in a macro file has 4 fields, the label, type, + constraint and the definition. + +Build set files have the file extension ``.bset`` and the package build +configuration files have the file extension of ``.cfg``. The ``sb-set-builder`` +command will search for *build sets* and the ``sb-builder`` commands works with +package build configuration files. + +Both types of configuration files use the ``#`` character as a comment +character. Anything after this character on the line is ignored. There is no +block comment. + +Source and Patches +------------------ + +The RTEMS Source Builder provides a flexible way to manage source. Source and +patches are declare in configurations file using the ``source`` and ``patch`` +directives. These are a single line containing a Universal Resource Location or +URL and can contain macros and shell expansions. The :ref:`prep` section +details the *source* and *patch* directives + +The URL can reference remote and local source and patch resources. The +following schemes are provided: + +``http``: + Remote access using the HTTP protocol. + +``https``: + Remote access using the Secure HTTP protocol. + +``ftp``: + Remote access using the FTP protocol. + +``git``: + Remote access to a GIT repository. + +``pm``: + Remote access to a patch management repository. + +``file``: + Local access to an existing source directory. + +HTTP, HTTPS, and FTP +~~~~~~~~~~~~~~~~~~~~ + +Remote access to TAR or ZIP files is provided using HTTP, HTTPS and FTP +protocols. The full URL provided is used to access the remote file including +any query components. The URL is parsed to extract the file component and the +local source directory is checked for that file. If the file is located locally +the remote file is not downloaded. Currently no other checks are made. If a +download fails you need to manually remove the file from the source directory +and start the build process again. + +The URL can contain macros. These are expanded before issuing the request to +download the file. The standard GNU GCC compiler source URL is: + +.. code-block:: auto + + %source set<1> gcc<2> ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/gcc-%{gcc_version}.tar.bz2 + +.. topic:: Items: + + 1. The ``%source`` command's set command sets the source. The first is set + and following sets are ignored. + + 2. The source is part of the ``gcc`` group. + +The type of compression is automatically detected from the file extension. The +supported compression formats are: + +``gz``: + GNU ZIP + +``bzip2``: + BZIP2 + +``zip``: + ZIP + +``xy``: + XY + +The output of the decompression tool is fed to the standard ``tar`` utility if +not a ZIP file and unpacked into the build directory. ZIP files are unpacked by +the decompression tool and all other files must be in the tar file format. + +The ``%source`` directive typically supports a single source file tar or zip +file. The ``set`` command is used to set the URL for a specific source +group. The first set command encountered is registered and any further set +commands are ignored. This allows you to define a base standard source location +and override it in build and architecture specific files. You can also add +extra source files to a group. This is typically done when a collection of +source is broken down in a number of smaller files and you require the full +package. The source's ``setup`` command must reside in the ``%prep:`` section +and it unpacks the source code ready to be built. + +If the source URL references the GitHub API server https://api.github.com/ a +tarball of the specified version is download. For example the URL for the +STLINK project on GitHub and version is: + +.. code-block:: auto + + %define stlink_version 3494c11 + %source set stlink https://api.github.com/repos/texane/stlink/texane-stlink-%{stlink_version}.tar.gz + +GIT +~~~ + +A GIT repository can be cloned and used as source. The GIT repository resides +in the 'source' directory under the ``git`` directory. You can edit, update and +use the repository as you normally do and the results will used to build the +tools. This allows you to prepare and test patches in the build environment the +tools are built in. The GIT URL only supports the GIT protocol. You can control +the repository via the URL by appending options and arguments to the GIT +path. The options are delimited by ``?`` and option arguments are delimited +from the options with ``=``. The options are: + +``protocol``: + Use a specific protocol. The supported values are ``ssh``, ``git``, ``http``, + ``https``, ``ftp``, ``ftps``, ``rsync``, and ``none``. + +``branch``: + Checkout the specified branch. + +``pull``: + Perform a pull to update the repository. + +``fetch``: + Perform a fetch to get any remote updates. + +``reset``: + Reset the repository. Useful to remove any local changes. You can pass the + ``hard`` argument to force a hard reset. + +An example is: + +.. code-block:: auto + + %source set gcc git://gcc.gnu.org/git/gcc.git?branch=gcc-4_7-branch?reset=hard + +This will clone the GCC git repository and checkout the 4.7-branch and perform +a hard reset. You can select specific branches and apply patches. The +repository is cleaned up before each build to avoid various version control +errors that can arise. + +The protocol option lets you set a specific protocol. The ``git://`` prefix +used by the RSB to select a git repository can be removed using *none* or +replaced with one of the standard git protcols. + +CVS +~~~ + +A CVS repository can be checked out. CVS is more complex than GIT to handle +because of the modules support. This can effect the paths the source ends up +in. The CVS URL only supports the CVS protocol. You can control the repository +via the URL by appending options and arguments to the CVS path. The options are +delimited by ``?`` and option arguments are delimited from the options with +``=``. The options are: + +``module``: + The module to checkout. + +``src-prefix``: + The path into the source where the module starts. + +``tag``: + The CVS tag to checkout. + +``date``: + The CVS date to checkout. + +The following is an example of checking out from a CVS repository: + +.. code-block:: auto + + %source set newlib cvs://pserver:anoncvs@sourceware.org/cvs/src?module=newlib?src-prefix=src + +Macros and Defaults +------------------- + +The RTEMS Source Builder uses tables of *macros* read in when the tool +runs. The initial global set of macros is called the *defaults*. These values +are read from a file called ``defaults.mc`` and modified to suite your +host. This host specific adaption lets the Source Builder handle differences in +the build hosts. + +Build set and configuration files can define new values updating and extending +the global macro table. For example builds are given a release number. This is +typically a single number at the end of the package name. For example:: + + %define release 1 + +Once defined if can be accessed in a build set or package configuration file +with:: + + %{release} + +The ``sb-defaults`` command lists the defaults for your host. I will not include +the output of this command because of its size:: + + $ ../source-builder/sb-defaults + +A nested build set is given a separate copy of the global macro maps. Changes +in one change set are not seen in other build sets. That same happens with +configuration files unless inline includes are used. Inline includes are seen +as part of the same build set and configuration and changes are global to that +build set and configuration. + +Macro Maps and Files +~~~~~~~~~~~~~~~~~~~~ + +Macros are read in from files when the tool starts. The default settings are +read from the defaults macro file called ``defaults.mc`` located in the top +level RTEMS Source Builder command directory. User macros can be read in at +start up by using the ``--macros`` command line option. + +The format for a macro in macro files is:: + + Name Type Attribute String + +where ``Name`` is a case insensitive macro name, the ``Type`` field is: + +``none``: + Nothing, ignore. + +``dir``: + A directory path. + +``exe``: + An executable path. + +``triplet``: + A GNU style architecture, platform, operating system string. + +the ``Attribute`` field is: + +``none``: + Nothing, ignore + +``required``: + The host check must find the executable or path. + +``optional``: + The host check generates a warning if not found. + +``override``: + Only valid outside of the ``global`` map to indicate this macro overrides the + same one in the ``global`` map when the map containing it is selected. + +``undefine``: + Only valid outside of the ``global`` map to undefine the macro if it exists + in the ``global`` map when the map containing it is selected. The ``global`` + map's macro is not visible but still exists. + +and the ``String`` field is a single or tripled multiline quoted string. The +'String' can contain references to other macros. Macro that loop are not +currently detected and will cause the tool to lock up. + +Maps are declared anywhere in the map using the map directive:: + + # Comments + [my-special-map] <1> + _host: none, override, 'abc-xyz' + multiline: none, override, '''First line, + second line, + and finally the last line''' + +.. topic:: Items: + + 1. The map is set to ``my-special-map``. + +Any macro defintions following a map declaration are placed in that map and the +default map is ``global`` when loading a file. Maps are selected in +configuration files by using the ``%select`` directive:: + + %select my-special-map + +Selecting a map means all requests for a macro first check the selected map and +if present return that value else the ``global`` map is used. Any new macros or +changes update only the ``global`` map. This may change in future releases so +please make sure you use the ``override`` attribute. + +The macro files specificed on the command line are looked for in the +``_configdir`` paths. See <> variable for details. Included +files need to add the ``%{_configdir}`` macro to the start of the file. + +Macro map files can include other macro map files using the ``%include`` +directive. The macro map to build *binutils*, *gcc*, *newlib*, *gdb* and +RTEMS from version control heads is:: + + # <1> + # Build all tool parts from version control head. + # + %include %{_configdir}/snapshots/binutils-head.mc + %include %{_configdir}/snapshots/gcc-head.mc + %include %{_configdir}/snapshots/newlib-head.mc + %include %{_configdir}/snapshots/gdb-head.mc + +.. topic:: Items: + + 1. The file is ``config/snapshots/binutils-gcc-newlib-gdb-head.mc``. + +The macro map defaults to ``global`` at the start of each included file and the +map setting of the macro file including the other macro files does not change. + +Personal Macros +~~~~~~~~~~~~~~~ + +When the tools start to run they will load personal macros. Personal macros are +in the standard format for macros in a file. There are two places personal +macros can be configured. The first is the environment variable +``RSB_MACROS``. If present the macros from the file the environment variable +points to are loaded. The second is a file called ``.rsb_macros`` in your home +directory. You need to have the environment variable ``HOME`` defined for this +work. + +Report Mailing +-------------- + +The build reports can be mailed to a specific email address to logging and +monitoring. Mailing requires a number of parameters to function. These are: + +- To mail address + +- From mail address + +- SMTP host + +.. _To Mail Address: + +The ``to`` mail address is taken from the macro ``%{_mail_tools_to}`` and the +default is *rtems-tooltestresults at rtems.org*. You can override the default +with a personal or user macro file or via the command line option +``--mail-to``. + +.. _From Mail Address: + +The ``from`` mail address is taken from: + +- GIT configuration + +- User ``.mailrc`` file + +- Command line + +If you have configured an email and name in git it will be used used. If you do +not a check is made for a ``.mailrc`` file. The environment variable ``MAILRC`` +is used if present else your home directory is check. If found the file is +scanned for the ``from`` setting:: + + set from="Foo Bar " + +You can also support a from address on the command line with the ``--mail-from`` +option. + +The SMTP host is taken from the macro ``%{_mail_smtp_host}`` and the +default is ``localhost``. You can override the default with a personal +or user macro file or via the command line option ``--smtp-host``. + +Build Set Files +--------------- + +Build set files lets you list the packages in the build set you are defining +and have a file extension of ``.bset``. Build sets can define macro variables, +inline include other files and reference other build set or package +configuration files. + +Defining macros is performed with the ``%define`` macro:: + + %define _target m32r-rtems4.11 + +Inline including another file with the ``%include`` macro continues processing +with the specified file returning to carry on from just after the include +point:: + + %include rtems-4.11-base.bset + +This includes the RTEMS 4.11 base set of defines and checks. The configuration +paths as defined by ``_configdir`` are scanned. The file extension is optional. + +You reference build set or package configuration files by placing the file name +on a single line:: + + tools/rtems-binutils-2.22-1 + +The ``_configdir`` path is scanned for ``tools/rtems-binutils-2.22-1.bset`` or +``tools/rtems-binutils-2.22-1.cfg``. Build set files take precedent over +package configuration files. If ``tools/rtems-binutils-2.22-1`` is a build set +a new instance of the build set processor is created and if the file is a +package configuration the package is built with the package builder. This all +happens once the build set file has finished being scanned. + +Configuration Control +--------------------- + +The RTEMS Souce Builder is designed to fit within most verification and +validation processes. All of the RTEMS Source Builder is source code. The +Python code is source and comes with a commercial friendly license. All +configuration data is text and can be read or parsed with standard text based +tools. + +File naming provides configuration management. A specific version of a package +is captured in a specific set of configuration files. The top level +configuration file referenced in a *build set* or passed to the ``sb-builder`` +command relates to a specific configuration of the package being built. For +example the RTEMS configuration file ``rtems-gcc-4.7.2-newlib-2.0.0-1.cfg`` +creates an RTEMS GCC and Newlib package where the GCC version is 4.7.2, the +Newlib version is 2.0.0, plus any RTEMS specific patches that related to this +version. The configuration defines the version numbers of the various parts +that make up this package:: + + %define gcc_version 4.7.2 + %define newlib_version 2.0.0 + %define mpfr_version 3.0.1 + %define mpc_version 0.8.2 + %define gmp_version 5.0.5 + +The package build options, if there are any are also defined:: + + %define with_threads 1 + %define with_plugin 0 + %define with_iconv 1 + +The generic configuration may provide defaults in case options are not +specified. The patches this specific version of the package requires can be +included:: + + Patch0: gcc-4.7.2-rtems4.11-20121026.diff + +Finally including the GCC 4.7 configuration script:: + + %include %{_configdir}/gcc-4.7-1.cfg + +The ``gcc-4.7-1.cfg`` file is a generic script to build a GCC 4.7 compiler with +Newlib. It is not specific to RTEMS. A bare no operating system tool set can be +built with this file. + +The ``-1`` part of the file names is a revision. The GCC 4.7 script maybe +revised to fix a problem and if this fix effects an existing script the file is +copied and given a ``-2`` revision number. Any dependent scripts referencing +the earlier revision number will not be effected by the change. This locks down +a specific configuration over time. + +Personal Configurations +----------------------- + +The RSB supports personal configurations. You can view the RTEMS support in the +``rtems`` directory as a private configuration tree that resides within the RSB +source. There is also the ``bare`` set of configurations. You can create your +own configurations away from the RSB source tree yet use all that the RSB +provides. + +To create a private configuration change to a suitable directory:: + + $ cd ~/work + $ mkdir test + $ cd test + $ mkdir config + +and create a ``config`` directory. Here you can add a new configuration or +build set file. The section 'Adding New Configurations' details how to add a +new confguration. + +New Configurations +------------------ + +This section describes how to add a new configuration to the RSB. We will add a +configuration to build the Device Tree Compiler. The Device Tree Compiler or +DTC is part of the Flattened Device Tree project and compiles Device Tree +Source (DTS) files into Device Tree Blobs (DTB). DTB files can be loaded by +operating systems and used to locate the various resources such as base +addresses of devices or interrupt numbers allocated to devices. The Device Tree +Compiler source code can be downloaded from http://www.jdl.com/software. The +DTC is supported in the RSB and you can find the configuration files under the +``bare/config`` tree. I suggest you have a brief look over these files. + +Layering by Including +~~~~~~~~~~~~~~~~~~~~~ + +Configurations can be layered using the ``%include`` directive. The user +invokes the outer layers which include inner layers until all the required +configuration is present and the package can be built. The outer layers can +provide high level details such as the version and the release and the inner +layers provide generic configuration details that do not change from one +release to another. Macro variables are used to provide the specific +configuration details. + +Configuration File Numbering +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration files have a number at the end. This is a release number for that +configuration and it gives us the ability to track a specific configuration for +a specific version. For example lets say the developers of the DTC package +change the build system from a single makefile to autoconf and automake between +version 1.3.0 and version 1.4.0. The configuration file used to build the +package would change have to change. If we did not number the configuration +files the ability to build 1.1.0, 1.2.0 or 1.3.0 would be lost if we update a +common configuration file to build an autoconf and automake version. For +version 1.2.0 the same build script can be used so we can share the same +configuration file between version 1.1.0 and version 1.2.0. An update to any +previous release lets us still build the package. + +Common Configuration Scripts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Common configuration scripts that are independent of version, platform and +architecture are useful to everyone. These live in the Source Builder's +configuration directory. Currently there are scripts to build binutils, expat, +DTC, GCC, GDB and libusb. These files contain the recipes to build these +package without the specific details of the versions or patches being +built. They expect to be wrapped by a configuration file that ties the package +to a specific version and optionally specific patches. + +DTC Example +~~~~~~~~~~~ + +We will be building the DTC for your host rather than a package for RTEMS. We +will create a file called ``source-builder/config/dtc-1-1.cfg``. This is a +common script that can be used to build a specific version using a general +recipe. The file name is ``dtc-1-1.cfg`` where the ``cfg`` extension indicates +this is a configuration file. The first ``1`` says this is for the major +release 1 of the package and the last ``1`` is the build configuration version. + +The file starts with some comments that detail the configuration. If there is +anything unusual about the configuration it is a good idea to add something in +the comments here. The comments are followed by a check for the release. In +this case if a release is not provided a default of 1 is used:: + + # + # DTC 1.x.x Version 1. + # + # This configuration file configure's, make's and install's DTC. + # + + %if %{release} == %{nil} + %define release 1 + %endif + +The next section defines some information about the package. It does not effect +the build and is used to annotate the reports. It is recommended this +information is kept updated and accurate:: + + Name: dtc-%{dtc_version}-%{_host}-%{release} + Summary: Device Tree Compiler v%{dtc_version} for target %{_target} on host %{_host} + Version: %{dtc_version} + Release: %{release} + URL: http://www.jdl.com/software/ + BuildRoot: %{_tmppath}/%{name}-root-%(%{__id_u} -n) + +The next section defines the source and any patches. In this case there is a +single source package and it can be downloaded using the HTTP protocol. The RSB +knows this is GZip'ped tar file. If more than one package is needed, add +them increasing the index. The ``gcc-4.8-1.cfg`` configuration contains +examples of more than one source package as well as conditionally including +source packages based on the outer configuration options:: + + # + # Source + # + %source set dtc http://www.jdl.com/software/dtc-v%{dtc_version}.tgz + +The remainder of the script is broken in to the various phases of a build. They +are: + +. Preperation +. Bulding +. Installing, and +. Cleaning + +Preparation is the unpacking of the source, applying any patches as well as any +package specific set ups. This part of the script is a standard Unix shell +script. Be careful with the use of ``%`` and ``$``. The RSB uses ``%`` while +the shell scripts use ``$``. + +A standard pattern you will observe is the saving of the build's top +directory. This is used instead of changing into a subdirectory and then +changing to the parent when finished. Some hosts will change in a subdirectory +that is a link however changing to the parent does not change back to the +parent of the link rather it changes to the parent of the target of the link +and that is something the RSB nor you can track easily. The RSB configuration +script's are a collection of various subtle issues so please ask if you are +unsure why something is being done a particular way. + +The preparation phase will often include source and patch setup commands. Outer +layers can set the source package and add patches as needed while being able to +use a common recipe for the build. Users can override the standard build and +supply a custom patch for testing using the user macro command line interface:: + + # + # Prepare the source code. + # + %prep + build_top=$(pwd) + + %source setup dtc -q -n dtc-v%{dtc_version} + %patch setup dtc -p1 + + cd ${build_top} + +The configuration file ``gcc-common-1.cfg`` is a complex example of source +preparation. It contains a number of source packages and patches and it +combines these into a single source tree for building. It uses links to map +source into the GCC source tree so GCC can be built using the *single source +tree* method. It also shows how to fetch source code from version +control. Newlib is taken directly from its CVS repository. + +Next is the building phase and for the DTC example this is simply a matter of +running ``make``. Note the use of the RSB macros for commands. In the case of +``%{__make}`` it maps to the correct make for your host. In the case of BSD +systems we need to use the BSD make and not the GNU make. + +If your package requires a configuration stage you need to run this before the +make stage. Again the GCC common configuration file provides a detailed example:: + + %build + build_top=$(pwd) + + cd dtc-v%{dtc_version} + + %{build_build_flags} + + %{__make} PREFIX=%{_prefix} + + cd ${build_top} + +You can invoke make with the macro ``%{?_smp_flags}`` as a command line +argument. This macro is controlled by the ``--jobs`` command line option and +the host CPU detection support in the RSB. If you are on a multicore host you +can increase the build speed using this macro. It also lets you disabled +building on multicores to aid debugging when testing. + +Next is the install phase. This phase is a little more complex because you may +be building a tar file and the end result of the build is never actually +installed into the prefix on the build host and you may not even have +permissions to perform a real install. Most packages install to the ``prefix`` +and the prefix is typically supplied via the command to the RSB or the +package's default is used. The default can vary depending on the host's +operating system. To install to a path that is not the prefix the ``DESTDIR`` +make variable is used. Most packages should honour the ``DISTDIR`` make +variables and you can typically specify it on the command line to make when +invoking the install target. This results in the package being installed to a +location that is not the prefix but one you can control. The RSB provides a +shell variable called ``SB_BUILD_ROOT`` you can use. In a build set where you +are building a number of packages you can collect all the built packages in a +single tree that is captured in the tar file. + +Also note the use of the macro ``%{__rmdir}``. The use of these macros allow +the RSB to vary specific commands based on the host. This can help on hosts +like Windows where bugs can effect the standard commands such as ``rm``. There +are many many macros to help you. You can find these listed in the +``defaults.mc`` file and in the trace output. If you are new to creating and +editing configurations learning these can take a little time:: + + %install + build_top=$(pwd) + + %{__rmdir} -rf $SB_BUILD_ROOT + + cd dtc-v%{dtc_version} + %{__make} DESTDIR=$SB_BUILD_ROOT PREFIX=%{_prefix} install + + cd ${build_top} + +Finally there is an optional clean section. The RSB will run this section if +``--no-clean`` has not been provided on the command line. The RSB does clean up +for you. + +Once we have the configuration files we can execute the build using the +``sb-builder`` command. The command will perform the build and create a tar file +in the ``tar`` directory:: + + $ ../source-builder/sb-builder --prefix=/usr/local \ + --log=log_dtc devel/dtc-1.2.0 + RTEMS Source Builder, Package Builder v0.2.0 + config: devel/dtc-1.2.0 + package: dtc-1.2.0-x86_64-freebsd9.1-1 + download: http://www.jdl.com/software/dtc-v1.2.0.tgz -> sources/dtc-v1.2.0.tgz + building: dtc-1.2.0-x86_64-freebsd9.1-1 + $ ls tar + dtc-1.2.0-x86_64-freebsd9.1-1.tar.bz2 + +If you want to have the package installed automatically you need to create a +build set. A build set can build one or more packages from their configurations +at once to create a single package. For example the GNU tools is typically seen +as binutils, GCC and GDB and a build set will build each of these packages and +create a single build set tar file or install the tools on the host into the +prefix path. + +The DTC build set file is called ``dtc.bset`` and contains:: + + # + # Build the DTC. + # + + %define release 1 + + devel/dtc-1.2.0.cfg + +To build this you can use something similar to:: + + $ ../source-builder/sb-set-builder --prefix=/usr/local --log=log_dtc \ + --trace --bset-tar-file --no-install dtc + RTEMS Source Builder - Set Builder, v0.2.0 + Build Set: dtc + config: devel/dtc-1.2.0.cfg + package: dtc-1.2.0-x86_64-freebsd9.1-1 + building: dtc-1.2.0-x86_64-freebsd9.1-1 + tarball: tar/x86_64-freebsd9.1-dtc-set.tar.bz2 + cleaning: dtc-1.2.0-x86_64-freebsd9.1-1 + Build Set: Time 0:00:02.865758 + $ ls tar + dtc-1.2.0-x86_64-freebsd9.1-1.tar.bz2 x86_64-freebsd9.1-dtc-set.tar.bz2 + +The build is for a FreeBSD host and the prefix is for user installed +packages. In this example I cannot let the source builder perform the install +because I never run the RSB with root priviledges so a build set or bset tar +file is created. This can then be installed using root priviledges. + +The command also supplies the ``--trace`` option. The output in the log file +will contain all the macros. + +Debugging +~~~~~~~~~ + +New configuration files require debugging. There are two types of +debugging. The first is debugging RSB script bugs. The ``--dry-run`` option is +used here. Suppling this option will result in most of the RSB processing to be +performed and suitable output placed in the log file. This with the ``--trace`` +option should help you resolve any issues. + +The second type of bug to fix are related to the execution of one of +phases. These are usually a mix of shell script bugs or package set up or +configuration bugs. Here you can use any normal shell script type debug +technique such as ``set +x`` to output the commands or ``echo`` +statements. Debugging package related issues may require you start a build with +the RSB and supply ``--no-clean`` option and then locate the build directories +and change directory into them and manually run commands until to figure what +the package requires. + +Scripting +--------- + +Configuration files specify how to build a package. Configuration files are +scripts and have a ``.cfg`` file extension. The script format is based loosely +on the RPM spec file format however the use and purpose in this tool does not +compare with the functionality and therefore the important features of the spec +format RPM needs and uses. + +The script language is implemented in terms of macros. The built-in list is: + +``%{}``: + Macro expansion with conditional logic. + +``%()``: + Shell expansion. + +``%prep``: + The source preparation shell commands. + +``%build``: + The build shell commands. + +``%install``: + The package install shell commands. + +``%clean``: + The package clean shell commands. + +``%include``: + Inline include another configuration file. + +``%name``: + The name of the package. + +``%summary``: + A brief package description. Useful when reporting about a build. + +``%release``: + The package release. A number that is the release as built by this tool. + +``%version``: + The package's version string. + +``%buildarch``: + The build architecture. + +``%source``: + Define a source code package. This macro has a number appended. + +``%patch``: + Define a patch. This macro has a number appended. + +``%hash``: + Define a checksum for a source or patch file. + +``%echo``: + Print the following string as a message. + +``%warning``: + Print the following string as a warning and continue. + +``%error``: + Print the following string as an error and exit. + +``%select``: + Select the macro map. If there is no map nothing is reported. + +``%define``: + Define a macro. Macros cannot be redefined, you must first undefine it. + +``%undefine``: + Undefine a macro. + +``%if``: + Start a conditional logic block that ends with a ``%endif``. + +``%ifn``: + Inverted start of a conditional logic block. + +``%ifarch``: + Test the architecture against the following string. + +``%ifnarch``: + Inverted test of the architecture + +``%ifos``: + Test the host operating system. + +``%else``: + Start the *else* conditional logic block. + +``%endfi``: + End the conditional logic block. + +``%bconf_with``: + Test the build condition *with* setting. This is the ``--with-*`` command + line option. + +``%bconf_without``: + Test the build condition *without* setting. This is the ``--without-*`` + command line option. + +Expanding +~~~~~~~~~ + +A macro can be ``%{string}`` or the equivalent of ``%string``. The following macro +expansions supported are: + +``%{string}``: + Expand the 'string' replacing the entire macro text with the text in the + table for the entry 'string . For example if 'var' is 'foo' then ``${var}`` + would become ``foo``. + +``%{expand: string}``: + Expand the 'string' and then use it as a ``string`` to the macro expanding + the macro. For example if ``foo`` is set to ``bar`` and ``bar`` is set to + ``foobar`` then ``%{expand:foo}`` would result in ``foobar``. Shell expansion + can also be used. + +``%{with string}``: + Expand the macro to ``1`` if the macro ``with_string`` is defined else expand + to ``0``. Macros with the name ``with_string`` can be define with command + line arguments to the RTEMS Source Builder commands. + +``%{defined string}``: + Expand the macro to ``1`` if a macro of name ``string`` is defined else + expand to '0'. + +``%{?string: expression}``: + Expand the macro to ``expression`` if a macro of name ``string`` is defined + else expand to ``%{nil}``. + +``%{!?string: expression}``: + Expand the macro to ``expression`` if a macro of name ``string`` is not + defined. If the macro is define expand to ``%{nil}``. + +``%(expression)``: + Expand the macro to the result of running the ``expression`` in a host + shell. It is assumed this is a Unix type shell. For example ``%(whoami)`` + will return your user name and ``%(date)`` will return the current date + string. + +.. _prep: + +%prep +~~~~~ + +The +%prep+ macro starts a block that continues until the next block macro. The +*prep* or preparation block defines the setup of the package's source and is a +mix of RTEMS Source Builder macros and shell scripting. The sequence is +typically +%source+ macros for source, +%patch+ macros to patch the source +mixed with some shell commands to correct any source issues:: + + <1> <2> <3> + %source setup gcc -q -c -T -n %{name}-%{version} + +.. topic:: Items: + + 1. The source group to set up. + + 2. The source's name. + + 3. The version of the source. + +The source set up are declared with the source ``set`` and ``add`` commands. For +example: + +.. code-block:: auto + + %source set gdb http://ftp.gnu.org/gnu/gdb/gdb-%{gdb_version}.tar.bz2 + +This URL is the primary location of the GNU GDB source code and the RTEMS +Source Builder can download the file from this location and by inspecting the +file extension use ``bzip2`` decompression with +tar+. When the ``%prep`` +section is processed a check of the local ``source`` directory is made to see +if the file has already been downloaded. If not found in the source cache +directory the package is downloaded from the URL. You can append other base +URLs via the command line option ``--url``. This option accepts a comma +delimited list of sites to try. + +You could optionally have a few source files that make up the package. For +example GNU's GCC was a few tar files for a while and it is now a single tar +file. Support for multiple source files can be conditionally implemented with +the following scripting: + +.. code-block:: auto + + %source set gcc ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/gcc-code-%{gcc_version}.tar.bz2 + %source add gcc ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/gcc-g++-%{gcc_version}.tar.bz2 + %source setup gcc -q -T -D -n gcc-%{gcc_version} + +Separate modules use separate source groups. The GNU GCC compiler for RTEMS +uses Newlib, MPFR, MPC, and GMP source packages. You define the source with: + +.. code-block:: auto + + %source set gcc ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/gcc-%{gcc_version}.tar.bz2 + %source set newlib ftp://sourceware.org/pub/newlib/newlib-%{newlib_version}.tar.gz + %source set mpfr http://www.mpfr.org/mpfr-%{mpfr_version}/mpfr-%{mpfr_version}.tar.bz2 + %source set mpc http://www.multiprecision.org/mpc/download/mpc-%{mpc_version}.tar.gz + %source set gmp ftp://ftp.gnu.org/gnu/gmp/gmp-%{gmp_version}.tar.bz2 + +and set up with: + +.. code-block:: auto + + %source setup gcc -q -n gcc-%{gcc_version} + %source setup newlib -q -D -n newlib-%{newlib_version} + %source setup mpfr -q -D -n mpfr-%{mpfr_version} + %source setup mpc -q -D -n mpc-%{mpc_version} + %source setup gmp -q -D -n gmp-%{gmp_version} + +Patching also occurs during the preparation stage. Patches are handled in a +similar way to the source packages except you only ``add`` patches. Patches are +applied using the +setup+ command. The +setup+ command takes the default patch +option. You can provide options with each patch by adding them as arguments +before the patch URL. Patches with no options uses the +setup+ default. + +.. code-block:: auto + + %patch add gdb %{rtems_gdb_patches}/gdb-sim-arange-inline.diff + %patch add gdb -p0 <1> %{rtems_gdb_patches}/gdb-sim-cgen-inline.diff + +.. topic:: Items: + + 1. This patch has a custom option. + +To apply these patches:: + + %patch setup gdb -p1 <1> + +.. topic:: Items: + + 1. The default options. + +.. _build: + +%build +~~~~~~ + +The ``%build`` macro starts a block that continues until the next block +macro. The build block is a series of shell commands that execute to build the +package. It assumes all source code has been unpacked, patch and adjusted so +the build will succeed. + +The following is an example take from the GitHub STLink project. The STLink is +a JTAG debugging device for the ST ARM family of processors:: + + %build + export PATH="%{_bindir}:${PATH}" <1> + + cd texane-stlink-%{stlink_version} <2> + + ./autogen.sh <3> + + %if "%{_build}" != "%{_host}" + CFLAGS_FOR_BUILD="-g -O2 -Wall" \ <4> + %endif + CPPFLAGS="-I $SB_TMPPREFIX/include/libusb-1.0" \ <5> + CFLAGS="$SB_OPT_FLAGS" \ + LDFLAGS="-L $SB_TMPPREFIX/lib" \ + ./configure \ <6> + --build=%{_build} --host=%{_host} \ + --verbose \ + --prefix=%{_prefix} --bindir=%{_bindir} \ + --exec-prefix=%{_exec_prefix} \ + --includedir=%{_includedir} --libdir=%{_libdir} \ + --mandir=%{_mandir} --infodir=%{_infodir} + + %{__make} %{?_smp_mflags} all <7> + + cd .. + +.. topic:: Items: + + 1. Setup the PATH environment variable. This is not always needed. + + 2. This package builds in the source tree so enter it. + + 3. The package is actually checked directly out from the github project and + so it needs its autoconf and automake files generated. + + 4. Flags for a cross-compiled build. + + 5. Various settings passed to configure to customise the build. In this + example an include path is being set to the install point of + ``libusb``. This package requires ``libusb`` is built before it. + + 6. The ``configure`` command. The RTEMS Source Builder provides all the + needed paths as macro variables. You just need to provide them to + ``configure``. + + 7. Running make. Do not use ``make`` directly, use the RTEMS Source Builder's + defined value. This value is specific to the host. A large number of + packages need GNU make and on BSD systems this is ``gmake``. You can + optionally add the SMP flags if the packages build system can handle + parallel building with multiple jobs. The ``_smp_mflags`` value is + automatically setup for SMP hosts to match the number of cores the host + has. + +%install +~~~~~~~~ + +The ``%install`` macro starts a block that continues until the next block +macro. The install block is a series of shell commands that execute to install +the package. You can assume the package has built correctly when this block +starts executing. + +Never install the package to the actual *prefix* the package was built +with. Always install to the RTEMS Source Builder's temporary path defined in +the macro variable ``__tmpdir``. The RTEMS Source Builder sets up a shell +environment variable called ``SB_BUILD_ROOT`` as the standard install point. Most +packages support adding ``DESTDIR=`` to the ``make install`` command. + +Looking at the same example as in :ref:`build`:: + + %install + export PATH="%{_bindir}:${PATH}" <1> + rm -rf $SB_BUILD_ROOT <2> + + cd texane-stlink-%{stlink_version} <3> + %{__make} DESTDIR=$SB_BUILD_ROOT install <4> + + cd .. + +.. topic:: Items: + + 1. Setup the PATH environment variable. This is not always needed. + + 2. Clean any installed files. This makes sure the install is just what the + package installs and not any left over files from a broken build or + install. + + 3. Enter the build directory. In this example it just happens to be the + source directory. + + 4. Run ``make install`` to install the package overriding the ``DESTDIR`` + make variable. + +%clean +~~~~~~ + +The ``%clean`` macro starts a block that continues until the next block +macro. The clean block is a series of shell commands that execute to clean up +after a package has been built and install. This macro is currenly not been +used because the RTEMS Source Builder automatically cleans up. + +%include +~~~~~~~~ + +The ``%include`` macro inline includes the specific file. The ``__confdir`` +path is searched. Any relative path component of the include file is appended +to each part of the ``__configdir``. Adding an extension is optional as files +with ``.bset`` and ``.cfg`` are automatically searched for. + +Inline including means the file is processed as part of the configuration at +the point it is included. Parsing continues from the next line in the +configuration file that contains the ``%include`` macro. + +Including files allow a kind of configuration file reuse. The outer +configuration files provide specific information such as package version +numbers and patches and then include a generic configuration script which +builds the package:: + + %include %{_configdir}/gcc-4.7-1.cfg + +%name +~~~~~ + +The name of the package being built. The name typically contains the components +of the package and their version number plus a revision number. For the GCC +with Newlib configuration the name is typically:: + + Name: %{_target}-gcc-%{gcc_version}-newlib-%{newlib_version}-%{release} + +%summary +~~~~~~~~ + +The ``%summary`` is a brief description of the package. It is useful when +reporting. This information is not capture in the package anywhere. For the GCC +with Newlib configuration the summary is typically:: + + Summary: GCC v%{gcc_version} and Newlib v%{newlib_version} for target %{_target} on host %{_host} + +%release +~~~~~~~~ + +The ``%release`` is a packaging number that allows revisions of a package to +happen where no package versions change. This value typically increases when +the configuration building the package changes:: + + %define release 1 + +%version +~~~~~~~~ + +The ``%version`` macro sets the version the package. If the package is a single +component it tracks that component's version number. For example in the +``libusb`` configuration the ``%version`` is the same as ``%libusb_version``, +however in a GCC with Newlib configuration there is no single version +number. In this case the GCC version is used:: + + Version: %{gcc_version} + +%buildarch +~~~~~~~~~~ + +The ``%buildarch`` macro is set to the architecture the package contains. This +is currently not used in the RTEMS Source Builder and may go away. This macro +is more important in a real packaging system where the package could end up on +the wrong architecture. + +%source +~~~~~~~ + +The ``%source`` macro has 3 commands that controls what it does. You can +``set`` the source files, ``add`` source files to a source group, and ``setup`` +the source file group getting it ready to be used. + +Source files are source code files in tar or zip files that are unpacked, +copied or symbolically linked into the package's build tree. Building a package +requires one or more dependent packages. These are typically the packages +source code plus dependent libraries or modules. You can create any number of +these source groups and set each of them up with a separate source group for +each needed library or module. Each source group normally has a single tar, zip +or repository and the ``set`` defines this. Some projects split the source code +into separate tar or zip files and you install them by using the ``add`` +command. + +The first instance of a ``set`` command creates the source group and sets the +source files to be set up. Subsequent ``set`` commands for the same source +group are ignored. this lets you define the standard source files and override +them for specific releases or snapshots. To set a source file group: + +.. code-block:: auto + + %source set gcc <1> ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/gcc-%{gcc_version}.tar.bz2 + +.. topic:: Items: + + 1. The source group is ``gcc``. + +To add another source package to be installed into the same source tree you use +the ``add`` command: + +.. code-block:: auto + + %source add gcc ftp://ftp.gnu.org/gnu/gcc/gcc-%{gcc_version}/g++-%{gcc_version}.tar.bz2 + +The source ``setup`` command can only be issued in the ``%prep:`` section. The +setup is:: + + %source gcc setup -q -T -D -n %{name}-%{version} + +Accepted options are: + +``-n``: + The ``-n`` option is used to set the name of the software's build + directory. This is necessary only when the source archive unpacks into a + directory named other than ``-``. + +``-c``: + The ``-c`` option is used to direct ``%setup`` to create the top-level build + directory before unpacking the sources. + +``-D``: + The ``-D`` option is used to direct ``%setup`` to not delete the build + directory prior to unpacking the sources. This option is used when more than + one source archive is to be unpacked into the build directory, normally with + the ``-b`` or ``-a`` options. + +``-T``: + The ``-T`` option is used to direct %setup to not perform the default + unpacking of the source archive specified by the first ``Source:`` macro. It + is used with the ``-a`` or ``-b`` options. + +``-b ``: + The ``-b`` option is used to direct ``%setup`` to unpack the source archive + specified on the nth ``Source:`` macro line before changing directory into + the build directory. + +%patch +~~~~~~ + +The ``%patch`` macro has the same 3 command as the ``%source`` command however +the ``set`` commands is not really that useful with the ``%patch`` command. You +add patches with the ``add`` command and ``setup`` applies the patches. Patch +options can be added to each patch by placing them before the patch URL. If no +patch option is provided the default options passed to the ``setup`` command +are used. An option starts with a ``-``. The ``setup`` command must reside +inside the ``%prep`` section. + +Patches are grouped in a similar way to the ``%source`` macro so you can +control applying a group of patches to a specific source tree. + +The ``__patchdir`` path is searched. + +To add a patch:: + + %patch add gcc <1> gcc-4.7.2-rtems4.11-20121026.diff + %patch add gcc -p0 <2> gcc-4.7.2-rtems4.11-20121101.diff + +.. topic:: Items: + + 1. The patch group is ``gcc``. + + 2. Option for this specific patch. + +Placing ``%patch setup`` in the ``%prep`` section will apply the groups +patches:: + + %patch setup gcc <1> -p1 <2> + + 1. The patch group. + + 2. The default option used to apply the patch. + +%hash +~~~~~ + +The ``%hash`` macro requires 3 arguments and defines a checksum for a specific +file. The checksum is not applied until the file is checked before downloading +and once downloaded. A patch or source file that does not have a hash defined +generates a warning. + +A file to be checksummed must be unique in the source and patch directories. +The basename of the file is used as the key for the hash. + +The hash algorthim can be ``md5``, ``sha1``, ``sha224``, ``sha256``, +``sha384``, and ``sha512`` and we typically use ``md5``. + +To add a hash:: + + %hash md5 <1> net-snmp-%{net_snmp_version}.tar.gz <2> 7db683faba037249837b226f64d566d4 <3> + +.. topic:: Items: + + 1. The type of checksum. + + 2. The file to checksum. It can contain macros that are expanded for you. + + 3. The MD5 hash for the Net-SNMP file ``net-snmp-5.7.2.1.tar.gz``. + +Do not include a path with the file name. Only the basename is required. Files +can be searched for from a number of places and having a path conponent would +create confusion. This does mean files with hashes must be unique. + +Downloading off repositories such as git and cvs cannot be checksummed. It is +assumed those protocols and tools manage the state of the files. + +%echo +~~~~~ + +The ``%echo`` macro outputs the following string to stdout. This can also be used +as ``%{echo: message}``. + +%warning +~~~~~~~~ + +The ``%warning`` macro outputs the following string as a warning. This can also +be used as ``%{warning: message}``. + +%error +~~~~~~ + +The ``%error`` macro outputs the follow string as an error and exits the RTEMS +Source Builder. This can also be used as ``%{error: message}``. + +%select +~~~~~~~ + +The ``%select`` macro selects the map specified. If there is no map no error or +warning is generated. Macro maps provide a simple way for a user to override +the settings in a configuration file without having to edit it. The changes are +recorded in the build report so they can be traced. + +Configurations use different maps so macro overrides can target a specific +package. + +The default map is ``global``:: + + %select gcc-4.8-snapshot <1> + %define one_plus_one 2 <2> + +.. topic:: Items: + + 1. The map switches to ``gcc-4.8-snapshot``. Any overrides in this map will + be used. + + 2. Defining macros only updates the ``global`` map and not the selected map. + +%define +~~~~~~~ + +The ``%define`` macro defines a new macro or updates an existing one. If no +value is given it is assumed to be ``1``:: + + %define foo bar + %define one_plus_one 2 + %define one <1> + +.. topic:: Items: + + 1. The macro _one_ is set to 1. + +%undefine +~~~~~~~~~ + +The ``%undefine`` macro removes a macro if it exists. Any further references to +it will result in an undefine macro error. + +%if +~~~ + +The ``%if`` macro starts a conditional logic block that can optionally have a +*else* section. A test follows this macro and can have the following operators: + +.. list-table:: + + * - **%{}** + - Check the macro is set or *true*, ie non-zero:: + + %if ${foo} + %warning The test passes, must not be empty or is non-zero + %else + %error The test fails, must be empty or zero + %endif + + * - **\!** + - The *not* operator inverts the test of the macro:: + + %if ! ${foo} + %warning The test passes, must be empty or zero + %else + %error The test fails, must not be empty or is non-zero + %endif + + * - **==** + - The left hand size must equal the right hand side. For example:: + + %define one 1 + %if ${one} == 1 + %warning The test passes + %else + %error The test fails + %endif + You can also check to see if a macro is empty:: + + %if ${nothing} == %{nil} + %warning The test passes + %else + %error The test fails + + * - **!=** + - The left hand size does not equal the right hand side. For example:: + + # + # Check a value not being equal. + # + %define one 1 + %if ${one} != 2 + %warning The test passes + %else + %error The test fails + %endif + # + # Check if a macro is set. + # + %if ${something} != %{nil} + %warning The test passes + %else + %error The test fails + %endif + + * - **>** + - The left hand side is numerically greater than the right hand side. + + * - **>** + - The left hand side is numerically greater than or equal to the + right hand side. + + * - **<** + - The left hand side is numerically less than the right hand side. + + * - **<=** + - The left hand side is numerically less than or equal to the + right hand side. + +%ifn +~~~~ + +The ``%ifn`` macro inverts the normal ``%if`` logic. It avoids needing to provide +empty *if* blocks followed by *else* blocks. It is useful when checking if a +macro is defined:: + + %ifn %{defined foo} + %define foo bar + %endif + +%ifarch +~~~~~~~ + +The ``%ifarch`` is a short cut for ``%if %{_arch} == i386``. Currently not used. + +%ifnarch +~~~~~~~~ + +The ``%ifnarch`` is a short cut for ``%if %{_arch} != i386``. Currently not +used. + +%ifos +~~~~~ + +The ``%ifos`` is a short cut for ``%if %{_os} != mingw32``. It allows +conditional support for various operating system differences when building +packages. + +%else +~~~~~ + +The ``%else`` macro starts the conditional *else* block. + +%endfi +~~~~~~ + +The ``%endif`` macro ends a conditional logic block. + +%bconf_with +~~~~~~~~~~~ + +The ``%bconf_with`` macro provides a way to test if the user has passed a +specific option on the command line with the ``--with-