From fd6dc8c8de4dbc7ecf8a82a597cd5b43476fc8e3 Mon Sep 17 00:00:00 2001 From: Amar Takhar Date: Sun, 17 Jan 2016 19:19:43 -0500 Subject: Split document into seperate files by section. --- c_user/multiprocessing.rst | 539 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 539 insertions(+) create mode 100644 c_user/multiprocessing.rst (limited to 'c_user/multiprocessing.rst') diff --git a/c_user/multiprocessing.rst b/c_user/multiprocessing.rst new file mode 100644 index 0000000..dfd3324 --- /dev/null +++ b/c_user/multiprocessing.rst @@ -0,0 +1,539 @@ +Multiprocessing Manager +####################### + +.. index:: multiprocessing + +Introduction +============ + +In multiprocessor real-time systems, new +requirements, such as sharing data and global resources between +processors, are introduced. This requires an efficient and +reliable communications vehicle which allows all processors to +communicate with each other as necessary. In addition, the +ramifications of multiple processors affect each and every +characteristic of a real-time system, almost always making them +more complicated. + +RTEMS addresses these issues by providing simple and +flexible real-time multiprocessing capabilities. The executive +easily lends itself to both tightly-coupled and loosely-coupled +configurations of the target system hardware. In addition, +RTEMS supports systems composed of both homogeneous and +heterogeneous mixtures of processors and target boards. + +A major design goal of the RTEMS executive was to +transcend the physical boundaries of the target hardware +configuration. This goal is achieved by presenting the +application software with a logical view of the target system +where the boundaries between processor nodes are transparent. +As a result, the application developer may designate objects +such as tasks, queues, events, signals, semaphores, and memory +blocks as global objects. These global objects may then be +accessed by any task regardless of the physical location of the +object and the accessing task. RTEMS automatically determines +that the object being accessed resides on another processor and +performs the actions required to access the desired object. +Simply stated, RTEMS allows the entire system, both hardware and +software, to be viewed logically as a single system. + +Background +========== + +.. index:: multiprocessing topologies + +RTEMS makes no assumptions regarding the connection +media or topology of a multiprocessor system. The tasks which +compose a particular application can be spread among as many +processors as needed to satisfy the application’s timing +requirements. The application tasks can interact using a subset +of the RTEMS directives as if they were on the same processor. +These directives allow application tasks to exchange data, +communicate, and synchronize regardless of which processor they +reside upon. + +The RTEMS multiprocessor execution model is multiple +instruction streams with multiple data streams (MIMD). This +execution model has each of the processors executing code +independent of the other processors. Because of this +parallelism, the application designer can more easily guarantee +deterministic behavior. + +By supporting heterogeneous environments, RTEMS +allows the systems designer to select the most efficient +processor for each subsystem of the application. Configuring +RTEMS for a heterogeneous environment is no more difficult than +for a homogeneous one. In keeping with RTEMS philosophy of +providing transparent physical node boundaries, the minimal +heterogeneous processing required is isolated in the MPCI layer. + +Nodes +----- +.. index:: nodes, definition + +A processor in a RTEMS system is referred to as a +node. Each node is assigned a unique non-zero node number by +the application designer. RTEMS assumes that node numbers are +assigned consecutively from one to the ``maximum_nodes`` +configuration parameter. The node +number, node, and the maximum number of nodes, maximum_nodes, in +a system are found in the Multiprocessor Configuration Table. +The maximum_nodes field and the number of global objects, +maximum_global_objects, is required to be the same on all nodes +in a system. + +The node number is used by RTEMS to identify each +node when performing remote operations. Thus, the +Multiprocessor Communications Interface Layer (MPCI) must be +able to route messages based on the node number. + +Global Objects +-------------- +.. index:: global objects, definition + +All RTEMS objects which are created with the GLOBAL +attribute will be known on all other nodes. Global objects can +be referenced from any node in the system, although certain +directive specific restrictions (e.g. one cannot delete a remote +object) may apply. A task does not have to be global to perform +operations involving remote objects. The maximum number of +global objects is the system is user configurable and can be +found in the maximum_global_objects field in the Multiprocessor +Configuration Table. The distribution of tasks to processors is +performed during the application design phase. Dynamic task +relocation is not supported by RTEMS. + +Global Object Table +------------------- +.. index:: global objects table + +RTEMS maintains two tables containing object +information on every node in a multiprocessor system: a local +object table and a global object table. The local object table +on each node is unique and contains information for all objects +created on this node whether those objects are local or global. +The global object table contains information regarding all +global objects in the system and, consequently, is the same on +every node. + +Since each node must maintain an identical copy of +the global object table, the maximum number of entries in each +copy of the table must be the same. The maximum number of +entries in each copy is determined by the +maximum_global_objects parameter in the Multiprocessor +Configuration Table. This parameter, as well as the +maximum_nodes parameter, is required to be the same on all +nodes. To maintain consistency among the table copies, every +node in the system must be informed of the creation or deletion +of a global object. + +Remote Operations +----------------- +.. index:: MPCI and remote operations + +When an application performs an operation on a remote +global object, RTEMS must generate a Remote Request (RQ) message +and send it to the appropriate node. After completing the +requested operation, the remote node will build a Remote +Response (RR) message and send it to the originating node. +Messages generated as a side-effect of a directive (such as +deleting a global task) are known as Remote Processes (RP) and +do not require the receiving node to respond. + +Other than taking slightly longer to execute +directives on remote objects, the application is unaware of the +location of the objects it acts upon. The exact amount of +overhead required for a remote operation is dependent on the +media connecting the nodes and, to a lesser degree, on the +efficiency of the user-provided MPCI routines. + +The following shows the typical transaction sequence +during a remote application: + +# The application issues a directive accessing a + remote global object. + +# RTEMS determines the node on which the object + resides. + +# RTEMS calls the user-provided MPCI routine + GET_PACKET to obtain a packet in which to build a RQ message. + +# After building a message packet, RTEMS calls the + user-provided MPCI routine SEND_PACKET to transmit the packet to + the node on which the object resides (referred to as the + destination node). + +# The calling task is blocked until the RR message + arrives, and control of the processor is transferred to another + task. + +# The MPCI layer on the destination node senses the + arrival of a packet (commonly in an ISR), and calls the``rtems_multiprocessing_announce`` + directive. This directive readies the Multiprocessing Server. + +# The Multiprocessing Server calls the user-provided + MPCI routine RECEIVE_PACKET, performs the requested operation, + builds an RR message, and returns it to the originating node. + +# The MPCI layer on the originating node senses the + arrival of a packet (typically via an interrupt), and calls the RTEMS``rtems_multiprocessing_announce`` directive. This directive + readies the Multiprocessing Server. + +# The Multiprocessing Server calls the user-provided + MPCI routine RECEIVE_PACKET, readies the original requesting + task, and blocks until another packet arrives. Control is + transferred to the original task which then completes processing + of the directive. + +If an uncorrectable error occurs in the user-provided +MPCI layer, the fatal error handler should be invoked. RTEMS +assumes the reliable transmission and reception of messages by +the MPCI and makes no attempt to detect or correct errors. + +Proxies +------- +.. index:: proxy, definition + +A proxy is an RTEMS data structure which resides on a +remote node and is used to represent a task which must block as +part of a remote operation. This action can occur as part of the``rtems_semaphore_obtain`` and``rtems_message_queue_receive`` directives. If the +object were local, the task’s control block would be available +for modification to indicate it was blocking on a message queue +or semaphore. However, the task’s control block resides only on +the same node as the task. As a result, the remote node must +allocate a proxy to represent the task until it can be readied. + +The maximum number of proxies is defined in the +Multiprocessor Configuration Table. Each node in a +multiprocessor system may require a different number of proxies +to be configured. The distribution of proxy control blocks is +application dependent and is different from the distribution of +tasks. + +Multiprocessor Configuration Table +---------------------------------- + +The Multiprocessor Configuration Table contains +information needed by RTEMS when used in a multiprocessor +system. This table is discussed in detail in the section +Multiprocessor Configuration Table of the Configuring a System +chapter. + +Multiprocessor Communications Interface Layer +============================================= + +The Multiprocessor Communications Interface Layer +(MPCI) is a set of user-provided procedures which enable the +nodes in a multiprocessor system to communicate with one +another. These routines are invoked by RTEMS at various times +in the preparation and processing of remote requests. +Interrupts are enabled when an MPCI procedure is invoked. It is +assumed that if the execution mode and/or interrupt level are +altered by the MPCI layer, that they will be restored prior to +returning to RTEMS... index:: MPCI, definition + +The MPCI layer is responsible for managing a pool of +buffers called packets and for sending these packets between +system nodes. Packet buffers contain the messages sent between +the nodes. Typically, the MPCI layer will encapsulate the +packet within an envelope which contains the information needed +by the MPCI layer. The number of packets available is dependent +on the MPCI layer implementation... index:: MPCI entry points + +The entry points to the routines in the user’s MPCI +layer should be placed in the Multiprocessor Communications +Interface Table. The user must provide entry points for each of +the following table entries in a multiprocessor system: + +- initialization initialize the MPCI + +- get_packet obtain a packet buffer + +- return_packet return a packet buffer + +- send_packet send a packet to another node + +- receive_packet called to get an arrived packet + +A packet is sent by RTEMS in each of the following situations: + +- an RQ is generated on an originating node; + +- an RR is generated on a destination node; + +- a global object is created; + +- a global object is deleted; + +- a local task blocked on a remote object is deleted; + +- during system initialization to check for system consistency. + +If the target hardware supports it, the arrival of a +packet at a node may generate an interrupt. Otherwise, the +real-time clock ISR can check for the arrival of a packet. In +any case, the``rtems_multiprocessing_announce`` directive must be called +to announce the arrival of a packet. After exiting the ISR, +control will be passed to the Multiprocessing Server to process +the packet. The Multiprocessing Server will call the get_packet +entry to obtain a packet buffer and the receive_entry entry to +copy the message into the buffer obtained. + +INITIALIZATION +-------------- + +The INITIALIZATION component of the user-provided +MPCI layer is called as part of the ``rtems_initialize_executive`` +directive to initialize the MPCI layer and associated hardware. +It is invoked immediately after all of the device drivers have +been initialized. This component should be adhere to the +following prototype:.. index:: rtems_mpci_entry + +.. code:: c + + rtems_mpci_entry user_mpci_initialization( + rtems_configuration_table \*configuration + ); + +where configuration is the address of the user’s +Configuration Table. Operations on global objects cannot be +performed until this component is invoked. The INITIALIZATION +component is invoked only once in the life of any system. If +the MPCI layer cannot be successfully initialized, the fatal +error manager should be invoked by this routine. + +One of the primary functions of the MPCI layer is to +provide the executive with packet buffers. The INITIALIZATION +routine must create and initialize a pool of packet buffers. +There must be enough packet buffers so RTEMS can obtain one +whenever needed. + +GET_PACKET +---------- + +The GET_PACKET component of the user-provided MPCI +layer is called when RTEMS must obtain a packet buffer to send +or broadcast a message. This component should be adhere to the +following prototype: +.. code:: c + + rtems_mpci_entry user_mpci_get_packet( + rtems_packet_prefix \**packet + ); + +where packet is the address of a pointer to a packet. +This routine always succeeds and, upon return, packet will +contain the address of a packet. If for any reason, a packet +cannot be successfully obtained, then the fatal error manager +should be invoked. + +RTEMS has been optimized to avoid the need for +obtaining a packet each time a message is sent or broadcast. +For example, RTEMS sends response messages (RR) back to the +originator in the same packet in which the request message (RQ) +arrived. + +RETURN_PACKET +------------- + +The RETURN_PACKET component of the user-provided MPCI +layer is called when RTEMS needs to release a packet to the free +packet buffer pool. This component should be adhere to the +following prototype: +.. code:: c + + rtems_mpci_entry user_mpci_return_packet( + rtems_packet_prefix \*packet + ); + +where packet is the address of a packet. If the +packet cannot be successfully returned, the fatal error manager +should be invoked. + +RECEIVE_PACKET +-------------- + +The RECEIVE_PACKET component of the user-provided +MPCI layer is called when RTEMS needs to obtain a packet which +has previously arrived. This component should be adhere to the +following prototype: +.. code:: c + + rtems_mpci_entry user_mpci_receive_packet( + rtems_packet_prefix \**packet + ); + +where packet is a pointer to the address of a packet +to place the message from another node. If a message is +available, then packet will contain the address of the message +from another node. If no messages are available, this entry +packet should contain NULL. + +SEND_PACKET +----------- + +The SEND_PACKET component of the user-provided MPCI +layer is called when RTEMS needs to send a packet containing a +message to another node. This component should be adhere to the +following prototype: +.. code:: c + + rtems_mpci_entry user_mpci_send_packet( + uint32_t node, + rtems_packet_prefix \**packet + ); + +where node is the node number of the destination and packet is the +address of a packet which containing a message. If the packet cannot +be successfully sent, the fatal error manager should be invoked. + +If node is set to zero, the packet is to be +broadcasted to all other nodes in the system. Although some +MPCI layers will be built upon hardware which support a +broadcast mechanism, others may be required to generate a copy +of the packet for each node in the system. + +.. COMMENT: XXX packet_prefix structure needs to be defined in this document + +Many MPCI layers use the ``packet_length`` field of the``rtems_packet_prefix`` portion +of the packet to avoid sending unnecessary data. This is especially +useful if the media connecting the nodes is relatively slow. + +The ``to_convert`` field of the ``rtems_packet_prefix`` portion of the +packet indicates how much of the packet in 32-bit units may require conversion +in a heterogeneous system. + +Supporting Heterogeneous Environments +------------------------------------- +.. index:: heterogeneous multiprocessing + +Developing an MPCI layer for a heterogeneous system +requires a thorough understanding of the differences between the +processors which comprise the system. One difficult problem is +the varying data representation schemes used by different +processor types. The most pervasive data representation problem +is the order of the bytes which compose a data entity. +Processors which place the least significant byte at the +smallest address are classified as little endian processors. +Little endian byte-ordering is shown below: + +.. code:: c + + +---------------+----------------+---------------+----------------+ + | | | | | + | Byte 3 | Byte 2 | Byte 1 | Byte 0 | + | | | | | + +---------------+----------------+---------------+----------------+ + +Conversely, processors which place the most +significant byte at the smallest address are classified as big +endian processors. Big endian byte-ordering is shown below: +.. code:: c + + +---------------+----------------+---------------+----------------+ + | | | | | + | Byte 0 | Byte 1 | Byte 2 | Byte 3 | + | | | | | + +---------------+----------------+---------------+----------------+ + +Unfortunately, sharing a data structure between big +endian and little endian processors requires translation into a +common endian format. An application designer typically chooses +the common endian format to minimize conversion overhead. + +Another issue in the design of shared data structures +is the alignment of data structure elements. Alignment is both +processor and compiler implementation dependent. For example, +some processors allow data elements to begin on any address +boundary, while others impose restrictions. Common restrictions +are that data elements must begin on either an even address or +on a long word boundary. Violation of these restrictions may +cause an exception or impose a performance penalty. + +Other issues which commonly impact the design of +shared data structures include the representation of floating +point numbers, bit fields, decimal data, and character strings. +In addition, the representation method for negative integers +could be one’s or two’s complement. These factors combine to +increase the complexity of designing and manipulating data +structures shared between processors. + +RTEMS addressed these issues in the design of the +packets used to communicate between nodes. The RTEMS packet +format is designed to allow the MPCI layer to perform all +necessary conversion without burdening the developer with the +details of the RTEMS packet format. As a result, the MPCI layer +must be aware of the following: + +- All packets must begin on a four byte boundary. + +- Packets are composed of both RTEMS and application data. All RTEMS data + is treated as 32-bit unsigned quantities and is in the first ``to_convert`` + 32-bit quantities of the packet. The ``to_convert`` field is part of the``rtems_packet_prefix`` portion of the packet. + +- The RTEMS data component of the packet must be in native + endian format. Endian conversion may be performed by either the + sending or receiving MPCI layer. + +- RTEMS makes no assumptions regarding the application + data component of the packet. + +Operations +========== + +Announcing a Packet +------------------- + +The ``rtems_multiprocessing_announce`` directive is called by +the MPCI layer to inform RTEMS that a packet has arrived from +another node. This directive can be called from an interrupt +service routine or from within a polling routine. + +Directives +========== + +This section details the additional directives +required to support RTEMS in a multiprocessor configuration. A +subsection is dedicated to each of this manager’s directives and +describes the calling sequence, related constants, usage, and +status codes. + +MULTIPROCESSING_ANNOUNCE - Announce the arrival of a packet +----------------------------------------------------------- +.. index:: announce arrival of package + +**CALLING SEQUENCE:** + +.. index:: rtems_multiprocessing_announce + +.. code:: c + + void rtems_multiprocessing_announce( void ); + +**DIRECTIVE STATUS CODES:** + +NONE + +**DESCRIPTION:** + +This directive informs RTEMS that a multiprocessing +communications packet has arrived from another node. This +directive is called by the user-provided MPCI, and is only used +in multiprocessor configurations. + +**NOTES:** + +This directive is typically called from an ISR. + +This directive will almost certainly cause the +calling task to be preempted. + +This directive does not generate activity on remote nodes. + +.. COMMENT: COPYRIGHT (c) 2014. + +.. COMMENT: On-Line Applications Research Corporation (OAR). + +.. COMMENT: All rights reserved. + -- cgit v1.2.3