path: root/services/nfsclient/README

RTEMS-NFS
=========

A NFS-V2 client implementation for the RTEMS real-time
executive.

Author: Till Straumann <strauman@slac.stanford.edu>, 2002

Copyright 2002, Stanford University and
                Till Straumann <strauman@slac.stanford.edu>

Stanford Notice
***************

Acknowledgement of sponsorship
* * * * * * * * * * * * * * * *
This software was produced by the Stanford Linear Accelerator Center,
Stanford University, under Contract DE-AC03-76SFO0515 with the Department
of Energy.


Contents
--------
I   Overview
  1) Performance
  2) Reference Platform / Test Environment

II  Usage
  1) Initialization
  2) Mounting Remote Server Filesystems
  3) Unmounting
  4) Unloading
  5) Dumping Information / Statistics

III Implementation Details
  1) RPCIOD
  2) NFS
  3) RTEMS Resources Used By NFS/RPCIOD
  4) Caveats & Bugs

IV  Licensing & Disclaimers

I  Overview
-----------

This package implements a simple non-caching NFS
client for RTEMS. Most of the system calls are
supported with the exception of 'mount', i.e. it
is not possible to mount another FS on top of NFS
(mostly because of the difficulty that arises when
mount points are deleted on the server). It
shouldn't be hard to do, though.

Note: this client supports NFS vers. 2 / MOUNT vers. 1;
      NFS Version 3 or higher are NOT supported.

The package consists of two modules: RPCIOD and NFS
itself.

 - RPCIOD is a UDP/RPC multiplexor daemon. It takes
   RPC requests from multiple local client threads,
   funnels them through a single socket to multiple
   servers and dispatches the replies back to the
   (blocked) requestor threads.
   RPCIOD does packet retransmission and handles
   timeouts etc.
   Note however, that it does NOT do any XDR
   marshalling - it is up to the requestor threads
   to do the XDR encoding/decoding. RPCIOD _is_ RPC
   specific, though, because its message dispatching
   is based on the RPC transaction ID.

 - The NFS package maps RTEMS filesystem calls
   to proper RPCs, it does the XDR work and
   hands marshalled RPC requests to RPCIOD.
   All of the calls are synchronous, i.e. they
   block until they get a reply.

1) Performance
- - - - - - - -
Performance sucks (due to the lack of
readahead/delayed write and caching). On a fast
(100Mb/s) ethernet, it takes about 20s to copy a
10MB file from NFS to NFS.  I found, however, that
vxWorks' NFS client doesn't seem to be any
faster...

Since there is no buffer cache with read-ahead
implemented, all NFS reads are synchronous RPC
calls. Every read operation involves sending a
request and waiting for the reply. As long as the
overhead (sending request + processing it on the
server) is significant compared to the time it
takes to transferring the actual data, increasing
the amount of data per request results in better
throughput. The UDP packet size limit imposes a
limit of 8k per RPC call, hence reading from NFS
in chunks of 8k is better than chunks of 1k [but
chunks >8k are not possible, i.e., simply not
honoured: read(a_nfs_fd, buf, 20000) returns
8192]. This is similar to the old linux days
(mount with rsize=8k).  You can let stdio take
care of the buffering or use 8k buffers with
explicit read(2) operations. Note that stdio
honours the file-system's st_blksize field
if newlib is compiled with HAVE_BLKSIZE defined.
In this case, stdio uses 8k buffers for files
on NFS transparently. The blocksize NFS 
reports can be tuned with a global variable
setting (see nfs.c for details).

Further increase of throughput can be achieved
with read-ahead (issuing RPC calls in parallel
[send out request for block n+1 while you are
waiting for data of block n to arrive]).  Since
this is not handled by the file system itself, you
would have to code this yourself e.g., using
parallel threads to read from a single file from
interleaved offsets.

Another obvious improvement can be achieved if
processing the data takes a significant amount of
time. Then, having a pipeline of threads for
reading data and processing them makes sense
[thread b processes chunk n while thread a blocks
in read(chunk n+1)].

Some performance figures:
Software: src/nfsTest.c:nfsReadTest() [data not
          processed in any way].
Hardware: MVME6100
Network:  100baseT-FD
Server:   Linux-2.6/RHEL4-smp [dell precision 420]
File:     10MB

Results:
Single threaded ('normal') NFS read, 1k buffers: 3.46s (2.89MB/s)
Single threaded ('normal') NFS read, 8k buffers: 1.31s (7.63MB/s)
Multi  threaded; 2 readers, 8k buffers/xfers:    1.12s (8.9 MB/s)  
Multi  threaded; 3 readers, 8k buffers/xfers:    1.04s (9.6 MB/s)

2) Reference Platform
- - - - - - - - - - -
RTEMS-NFS was developed and tested on

 o RTEMS-ss20020301 (local patches applied)
 o PowerPC G3, G4 on Synergy SVGM series board
   (custom 'SVGM' BSP, to be released soon)
 o PowerPC 604 on MVME23xx
   (powerpc/shared/motorola-powerpc BSP)
 o Test Environment:
    - RTEMS executable running CEXP
    - rpciod/nfs dynamically loaded from TFTPfs
    - EPICS application dynamically loaded from NFS;
      the executing IOC accesses all of its files
      on NFS.

II Usage
---------

After linking into the system and proper initialization
(rtems-NFS supports 'magic' module initialization when
loaded into a running system with the CEXP loader),
you are ready for mounting NFSes from a server
(I avoid the term NFS filesystem because NFS already
stands for 'Network File System').

You should also read the

  - "RTEMS Resources Used By NFS/RPCIOD"
  - "CAVEATS & BUGS"

below.

1) Initialization
- - - - - - - - - 
NFS consists of two modules who must be initialized:

 a) the RPCIO daemon package; by calling

      rpcUdpInit();

    note that this step must be performed prior to
    initializing NFS:

 b) NFS is initialized by calling

      nfsInit( smallPoolDepth, bigPoolDepth );

    if you supply 0 (zero) values for the pool
    depths, the compile-time default configuration
    is used which should work fine.

NOTE: when using CEXP to load these modules into a
running system, initialization will be performed
automagically.

2) Mounting Remote Server Filesystems
- - - - - - - - - - - - - - - - - - -

There are two interfaces for mounting an NFS:

 - The (non-POSIX) RTEMS 'mount()' call:

     mount( &mount_table_entry_pointer,
            &filesystem_operations_table_pointer,
            options,
            device,
            mount_point )

    Note that you must specify a 'mount_table_entry_pointer'
    (use a dummy) - RTEMS' mount() doesn't grok a NULL for
    the first argument.

     o for the 'filesystem_operations_table_pointer', supply

         &nfs_fs_ops
   
     o options are constants (see RTEMS headers) for specifying
       read-only / read-write mounts.

     o the 'device' string specifies the remote filesystem
       who is to be mounted. NFS expects a string conforming
       to the following format (EBNF syntax):

         [ <uid> '.' <gid> '@' ] <hostip> ':' <path>

       The first optional part of the string allows you
       to specify the credentials to be used for all
       subsequent transactions with this server. If the
       string is omitted, the EUID/EGID of the executing
       thread (i.e. the thread performing the 'mount' - 
       NFS will still 'remember' these values and use them
       for all future communication with this server).
       
       The <hostip> part denotes the server IP address
       in standard 'dot' notation. It is followed by
       a colon and the (absolute) path on the server.
       Note that no extra characters or whitespace must
       be present in the string. Example 'device' strings
       are:

         "300.99@192.168.44.3:/remote/rtems/root"

         "192.168.44.3:/remote/rtems/root"

    o the 'mount_point' string identifies the local
      directory (most probably on IMFS) where the NFS
      is to be mounted. Note that the mount point must
      already exist with proper permissions.

 - Alternate 'mount' interface. NFS offers a more
   convenient wrapper taking three string arguments:

	nfsMount(uidgid_at_host, server_path, mount_point)

   This interface does DNS lookup (see reentrancy note
   below) and creates the mount point if necessary.
   
   o the first argument specifies the server and
     optionally the uid/gid to be used for authentication.
     The semantics are exactly as described above:

       [ <uid> '.' <gid> '@' ] <host>
     
     The <host> part may be either a host _name_ or
     an IP address in 'dot' notation. In the former
     case, nfsMount() uses 'gethostbyname()' to do
     a DNS lookup.

     IMPORTANT NOTE: gethostbyname() is NOT reentrant/
     thread-safe and 'nfsMount()' (if not provided with an
     IP/dot address string) is hence subject to race conditions.
 
   o the 'server_path' and 'mount_point' arguments
     are described above.
     NOTE: If the mount point does not exist yet,
           nfsMount() tries to create it.

   o if nfsMount() is called with a NULL 'uidgid_at_host'
     argument, it lists all currently mounted NFS

3) Unmounting
- - - - - - -
An NFS can be unmounted using RTEMS 'unmount()'
call (yep, it is unmount() - not umount()):

  unmount(mount_point)

Note that you _must_ supply the mount point (string
argument). It is _not_ possible to specify the
'mountee' when unmounting. NFS implements no
convenience wrapper for this (yet), essentially because
(although this sounds unbelievable) it is non-trivial
to lookup the path leading to an RTEMS filesystem
directory node.

4) Unloading
- - - - - - -
After unmounting all NFS from the system, the NFS
and RPCIOD modules may be stopped and unloaded.
Just call 'nfsCleanup()' and 'rpcUdpCleanup()'
in this order. You should evaluate the return value
of these routines which is non-zero if either
of them refuses to yield (e.g. because there are
still mounted filesystems).
Again, when unloading is done by CEXP this is
transparently handled.

5) Dumping Information / Statistics
- - - - - - - - - - - - - - - - - -

Rudimentary RPCIOD statistics are printed
to a file (stdout when NULL) by

  int rpcUdpStats(FILE *f)

A list of all currently mounted NFS can be
printed to a file (stdout if NULL) using

  int nfsMountsShow(FILE *f)

For convenience, this routine is also called
by nfsMount() when supplying NULL arguments.

III Implementation Details
--------------------------

1) RPCIOD
- - - - -

RPCIOD was created to

a) avoid non-reentrant librpc calls.
b) support 'asynchronous' operation over a single
   socket.

RPCIOD is a daemon thread handling 'transaction objects'
(XACTs) through an UDP socket.  XACTs are marshalled RPC
calls/replies associated with RPC servers and requestor
threads.

requestor thread:                 network:

       XACT                        packet  
        |                            |
        V                            V
  | message queue |              ( socket )
        |                            |  ^
        ---------->          <-----  |  |
                     RPCIOD             |
                   /       --------------
           timeout/         (re) transmission
                         

A requestor thread drops a transaction into 
the message queue and goes to sleep.  The XACT is
picked up by rpciod who is listening for events from
three sources:

  o the request queue
  o packet arrival at the socket
  o timeouts

RPCIOD sends the XACT to its destination server and
enqueues the pending XACT into an ordered list of
outstanding transactions.

When a packet arrives, RPCIOD (based on the RPC transaction
ID) looks up the matching XACT and wakes up the requestor
who can then XDR-decode the RPC results found in the XACT
object's buffer.

When a timeout expires, RPCIOD examines the outstanding
XACT that is responsible for the timeout. If its lifetime
has not expired yet, RPCIOD resends the request. Otherwise,
the XACT's error status is set and the requestor is woken up.

RPCIOD dynamically adjusts the retransmission intervals
based on the average round-trip time measured (on a per-server
basis).

Having the requestors event driven (rather than blocking
e.g. on a semaphore) is geared to having many different
requestors (one synchronization object per requestor would
be needed otherwise).

Requestors who want to do asynchronous IO need a different
interface which will be added in the future.

1.a) Reentrancy
- - - - - - - - 
RPCIOD does no non-reentrant librpc calls.

1.b) Efficiency
- - - - - - - - 
We shouldn't bother about efficiency until pipelining (read-ahead/
delayed write) and caching are implemented. The round-trip delay
associated with every single RPC transaction clearly is a big
performance killer.

Nevertheless, I could not withstand the temptation to eliminate
the extra copy step involved with socket IO:

A user data object has to be XDR encoded into a buffer. The 
buffer given to the socket where it is copied into MBUFs.
(The network chip driver might even do more copying).

Likewise, on reception 'recvfrom' copies MBUFS into a user
buffer which is XDR decoded into the final user data object.

Eliminating the copying into (possibly multiple) MBUFS by
'sendto()' is actually a piece of cake. RPCIOD uses the
'sosend()' routine [properly wrapped] supplying a single
MBUF header who directly points to the marshalled buffer
:-)

Getting rid of the extra copy on reception was (only a little)
harder: I derived a 'XDR-mbuf' stream from SUN's xdr_mem which
allows for XDR-decoding out of a MBUF chain who is obtained by
soreceive().

2) NFS
- - - -
The actual NFS implementation is straightforward and essentially
'passive' (no threads created). Any RTEMS task executing a
filesystem call dispatched to NFS (such as 'opendir()', 'lseek()'
or 'unlink()') ends up XDR encoding arguments, dropping a
XACT into RPCIOD's message queue and going to sleep.
When woken up by RPCIOD, the XACT is decoded (using the XDR-mbuf
stream mentioned above) and the properly cooked-up results are
returned.

3) RTEMS Resources Used By NFS/RPCIOD
- - - - - - - - - - - - - - - - - - -

The RPCIOD/NFS package uses the following resources. Some
parameters are compile-time configurable - consult the
source files for details.

RPCIOD:
 o 1 task 
 o 1 message queue
 o 1 socket/filedescriptor
 o 2 semaphores (a third one is temporarily created during
   rpcUdpCleanup()).
 o 1 RTEMS EVENT (by default RTEMS_EVENT_30).
   IMPORTANT: this event is used by _every_ thread executing
              NFS system calls and hence is RESERVED.
 o 3 events only used by RPCIOD itself, i.e. these must not
   be sent to RPCIOD by no other thread (except for the intended
   use, of course). The events involved are 1,2,3.
 o preemption disabled sections:      NONE
 o sections with interrupts disabled: NONE
 o NO 'timers' are used (timer code would run in IRQ context)
 o memory usage: n.a

NFS:
 o 2 message queues
 o 2 semaphores
 o 1 semaphore per mounted NFS
 o 1 slot in driver entry table (for major number)
 o preemption disabled sections:      NONE
 o sections with interrupts disabled: NONE
 o 1 task + 1 semaphore temporarily created when
   listing mounted filesystems (rtems_filesystem_resolve_location())

4) CAVEATS & BUGS
- - - - - - - - -
Unfortunately, some bugs crawl around in the filesystem generics.
(Some of them might already be fixed in versions later than
rtems-ss-20020301).
I recommend to use the patch distributed with RTEMS-NFS.

 o RTEMS uses/used (Joel said it has been fixed already) a 'short'
   ino_t which is not enough for NFS.
   The driver detects this problem and enables a workaround. In rare
   situations (mainly involving 'getcwd()' improper inode comparison
   may result (due to the restricted size, stat() returns st_ino modulo
   2^16). In most cases, however, st_dev is compared along with st_ino
   which will give correct results (different files may yield identical
   st_ino but they will have different st_dev). However, there is 
   code (in getcwd(), for example) who assumes that files residing
   in one directory must be hosted by the same device and hence omits
   the st_dev comparison. In such a case, the workaround will fail.
 
   NOTE: changing the size (sys/types.h) of ino_t from 'short' to 'long'
         is strongly recommended. It is NOT included in the patch, however
         as this is a major change requiring ALL of your sources to
         be recompiled.

   THE ino_t SIZE IS FIXED IN GCC-3.2/NEWLIB-1.10.0-2 DISTRIBUTED BY
   OAR.

 o You may work around most filesystem bugs by observing the following
   rules:

    * never use chroot() (fixed by the patch)
    * never use getpwent(), getgrent() & friends - they are NOT THREAD
      safe (fixed by the patch)
    * NEVER use rtems_libio_share_private_env() - not even with the
      patch applied. Just DONT - it is broken by design.
    * All threads who have their own userenv (who have called
      rtems_libio_set_private_env()) SHOULD 'chdir("/")' before
      terminating. Otherwise, (i.e. if their cwd is on NFS), it will
      be impossible to unmount the NFS involved.

 o The patch slightly changes the semantics of 'getpwent()' and
   'getgrent()' & friends (to what is IMHO correct anyways - the patch is
   also needed to fix another problem, however): with the patch applied,
   the passwd and group files are always accessed from the 'current' user
   environment, i.e. a thread who has changed its 'root' or 'uid' might
   not be able to access these files anymore.
      
 o NOTE: RTEMS 'mount()' / 'unmount()' are NOT THREAD SAFE.

 o The NFS protocol has no 'append' or 'seek_end' primitive. The client
   must query the current file size (this client uses cached info) and
   change the local file pointer accordingly (in 'O_APPEND' mode).
   Obviously, this involves a race condition and hence multiple clients
   writing the same file may lead to corruption.

IV Licensing & Disclaimers
--------------------------

NFS is distributed under the SLAC License - consult the
separate 'LICENSE' file.

Government disclaimer of liability
- - - - - - - - - - - - - - - - -
Neither the United States nor the United States Department of Energy,
nor any of their employees, makes any warranty, express or implied,
or assumes any legal liability or responsibility for the accuracy,
completeness, or usefulness of any data, apparatus, product, or process
disclosed, or represents that its use would not infringe privately
owned rights.

Stanford disclaimer of liability
- - - - - - - - - - - - - - - - -
Stanford University makes no representations or warranties, express or
implied, nor assumes any liability for the use of this software.

Maintenance of notice
- - - - - - - - - - -
In the interest of clarity regarding the origin and status of this
software, Stanford University requests that any recipient of it maintain
this notice affixed to any distribution by the recipient that contains a
copy or derivative of this software.