VxWorks on Nova
1 May 2000
This document details the state of the VxWorks installation on nova.fnal.gov.
The document will also explain the goals which lead to the current state
along with the administrative dealings that have occurred with Wind River
Systems regarding the nova VxWorks installation. This document deals
principally with the administration of Tornado 2.0/ VxWorks 5.4.
There are four installations of VxWorks on nova - Tornado 1.0.1 for 68k,
Tornado 2.0 for 68k, Tornado 1.0.1 for PowerPC,
and Tornado 2.0 for PowerPC.
The current state of VxWorks on nova was largely a result of trying to
satisfy the following goals. How each goal was achieved is
presented along with the presentation of the goal itself.
Minimize the modifications FNAL needs to make to the WRS installation,
but where modifications are required maintain a detailed history of such
modifications. The WRS installations are the 68k_1.0.1, 68k_2.0,
ppc_1.0.1, and ppc_2.0 subdirectories in /usr/local/vxworks/.
Thanks to Tornado 2.0's project facility the need to modify WRS code is
greatly reduced. Unfortunately, modifications like infinite retry
bootroms still required modification to WRS code. CVS
was used to keep track of such modifications. The fact that
the WRS distribution is not entirely source code and is large made this
difficult. So the four WRS install directories, were
converted into four CVS working directories. Each working directory
with the same root in the file system as the original install directory.
Initially each working directory and all its subdirectories contained only
a CVS directory generated by CVS and a .cvsignore file which listed all
the files in that subdirectory. At this point the directory
hierarchy of the installation was in CVS, but in fact none of the WRS files
were. But at this point it was ok that the files were not in
CVS, since they had not been modified. The first time a source file
is discovered that requires modification, it needs to be removed from the
.cvsignore file, added and committed to CVS before any mods are made, have
its permissions updated so it can be modified. The script /usr/local/vxworks/scripts/vwedit
will perform these steps on a single file specified as its only command
line argument. Once this script is executed on a file,
the file may be modified and committed as many times as one wants, knowing
that all modifications are being recorded in CVS.
One issue that is raised by the co-location of the install and working
directories is the application of WRS patches. This is handled as
follows - if the patch creates a new file add the file to the .cvsignore
file, if it modifies a file commit the change if the file was placed
in CVS if not then the mod does not need to be recorded because FNAL never
modified the file so no action is needed, if it removes a file if that
file was in CVS then remove the file from CVS if the file was not in CVS
no actions is needed. If the patch modifies or removes
a file that was in CVS, one should investigate the interaction of the patch
the FNAL modifications to the file. Several patches have already
been successfully applied to VxWorks on nova.
Provide standard downloadable VxWorks kernels for all major FE platforms
for use with the controls department's FE infrastructure libraries, i.e.,
MOOC, etc. In the case were the standard kernel is sufficient, standard
kernels allow FE programmers to get a FE up and running quicker.
Standard kernels also help insure that MOOC, etc. can assume a certain
level of functionality, e.g., that UDP multicast is supported.
The configuration and additional source used to build the standard kernels
should as much as possible be kept separate from the WRS install directory.
Fortunately, the project facility facilitates this well. Some FEs
will not be able to use the standard downloadable kernel for their platform
for valid reasons. For example, frig needs to burn its OS into ROM
so that it will still be able to control the frig system during a network
failure. Therefore, the building of custom kernels needs
to be allowed.
A "standard" vxworks 5.4 bootrom and vxworks 5.4 downloadable kernel are
provide for the mv162 (the downloadable kernel comes in a 4MB and 16MB
version), mv2303, mv2400, and mv2603.
The standard bootroms are available via TFTP from nova in the directory
vxworks/bootroms/ relative to the TFTP root. The bootroms are named
mv162_5.4.bin, mv2303_5.4.bin, mv2400_5.4.bin, and mv2603_5.4.bin.
These bootroms all have the infinite retry modification. Based
on experiment it appears a 5.4 bootrom requires a 5.4 downloadable kernel.
When these 5.4 bootroms were used to download a 5.3 kernel, the processor
hung just after displaying the "Starting at" message. No such problem
as ever occurred when loading a 5.4 kernel with the bootroms, except for
the now fixed mv2400 project facility bug. A processors flash
may be programmed with these bootroms in the standard way documented by
WRS. I called WRS to see if they would consider making infinite
retry a standard part of the distribution, this was assigned TSR 150977.
They were not interested.
The mv162 BSP installation directory was modified to correct the incorrect
oscillator frequency in sysLib.c. For whatever reason WRS as yet
to fix this. WRS has assigned this bug the SPR 9570.
This corrects all downloadable kernels built for the mv162 not just the
The configuration of the standard kernel up to this point has not required
modifications to the WRS install directories, thanks to the project facility.
The project facility allows the kernel configuration files to be kept outside
of the install directories. The configuration files for the
standard kernels are in CVS on nova rooted at vxworks/builds relative to
the root of the Beams Division Microprocessor's CVS repository, which is
at /export/cvs/bdmicrop on nova. The modules files contains
the shortcut vxworks_builds by which the standard build's configuration
files can be checked out. Unfortunately, the project facility
maintains the absolute pathname name of the location of the configuration
files, so when you check out the builds you will need to update the relatively
small number of times these absolute paths occur to reflect that the files
have been checked out in your area and not the last guy to commit a configuration
change. You should modify and view the current configuration
using the project facility, but once your changes are made you will need
to use CVS at the command line to commit your changes. The
root of this configuration files hierarchy contains a script and a Makefile
which calls it. The Makefile has an install target which will
copy all the VxWorks kernels into the appropriate place on fecode-bd so
that FEs may download them. Make sure a build was actually
done in the project facility before executing a make install.
All installed standard kernels are rooted at fecode-bd:vxworks_boot/kernel
relative to the vxworks_boot home directory. The kernel directory
contains a subdirectory for each of the five platforms. The subdirectories
are named mv162-16MB-mooc, mv162-4MB-mooc, mv2303-mooc, mv2400-mooc, and
mv2603-mooc. Within each platform specific directory there
are subdirectories for each revision of the standard kernel.
These revision directories contain a vxworks kernel, vxWorks, and its symbol
table, vxWorks.sym. The revision directories are named with
the GMT time that the revision was installed. The format of the time
is <year>-<month>-<day of month>-<time in 24 hour style>.
The install target does not touch old revisions.
The standard kernels have made many configuration choices for the FE programmer
which you may view from the project facility. The general rule
was to include any reasonable functionality. Some of the more
significant configuration options included are C++, spy (except for mv2400),
startup scripts, show routines, downloaded symbol table, DNS, security
w/ the standard vxworks_boot login, SNTP, NFS client, and ZBUF sockets.
In addition the project facility and vxworks allow each project, i.e.,
downloadable kernel, to provide a custom routine, usrAppInit(), defined
in usrAppInit.c. The standard kernels provide additional functionality
by modifying usrAppInit() to call the routine vwsInit(). vwsInit()
is defined by libvws.o which is an FNAL library with a collection of routines
to augment basic vxWorks functionality. libvws.o is installed
on fecode-bd in the modules area for each processor. (Since
libvws.o is actually linked in and not dynamically loaded it should be
in the lib directory, but that is another story.) To
have the kernel build link with libvws.o the EXTRA_MODULES parameter was
set appropriately in the project facility. The libvws.o source code
is the same for all processors and is in CVS on CRUSHER at fermi/fe_products/vwsupport.
The libvws.o code should be moved to nova when other crusher-based code
is moved to nova. The modules file provides the shorthand vwsupport.
The libvws.o support code was added as an EXTRA_MODULES instead of being
actually placed in with the kernel code so that libvws.o's source would
not be needlessly duplicated in each standard kernel area and so it would
benefit from fe.mk. One of the things that the libvws.o code needs
to do is retrieve the gateway specified in the boot parameters, so that
it can setup the routing tables with a default route to the gateway.
Unfortunately, WRS does not provide a BSP-independent way to the bootline
address. So each BSP was augmented with the routine vwsBootlineGet()
defined in vwsBootlineGet.c. This routine provides a BSP-independent
way for libvws.o to get the bootline address.
vwsInit() and its associated support routines in libvws.o provide some
basic functionality which seems best placed in the kernel.
In particular, vwsInit() provides better time-of-day and routing support.
vwsInit()'s time-of-day(TOD) support handles the initialization of TOD
on the VxWorks platform by retrieving the TOD via NTP from cns4.fnal.gov.
vwsInit() also initializes the TIMEZONE environment variable so that the
platform can provide the correct local time including daylight saving time
adjustments. (See ansiTime entry of library section of VxWorks
reference manual for details about the TIMEZONE environment variable.)
In order to insure that a platform's TOD stays reasonably accurate, which
has been a problem with WRS's BSP, and that the TIMEZONE is updated for
each year, since DST start and end dates vary each year, vwsInit() also
spawns the low priority task tTodTask, tTodTask updates the TOD every
hour via NTP from cns4. It updates the TIMEZONE as needed
also. The initialization of TOD, although not the periodic synchronization
or setting of the TIMEZONE environment variable, has typical been provided
by the acnet protocol library. Ideally, I think this
is best done by the kernel itself, which is why this functionality was
put here, so that acnet could concern itself only with the implementation
of the acnet protocol.
vwsInit()'s routing support handles the initialization of a platform's
routing table and subnet mask. The goal is to minimize the
number of entries in the platform's ARP tables, which is important to front-ends
like TLG that communicate with many hosts on the control system.
The number of entries is minimized by setting the subnet mask to 0xFFFFFF00
and establishing a default route via the gateway specified in the boot
Project Facility Issues
One will notice that the vxworks/builds directory in CVS contains two workspaces
and several projects. This is somewhat annoying since these
builds are intended to be the same configuration just for different platforms.
If one wants to add a kernel option to the standard build one must update
the configuration for each project, which is sure to introduce consistency
problems. Unfortunately, this is the best one can do with the
project facility. If the WRS environment variables on the host
that select a processor do not match the processor of the selected workspace,
project will issue a warning. So a workspace is needed for both processors.
A separate project is needed for each board because the Makefiles used
by the project facility use a define, BSP_DIR, which can only reference
a single BSP. Also some boards/BSPs do not support some options
which one would like in the standard kernel where possible, e.g., spy.
Even the 4MB and 16MB mv162 kernels required different projects, because
memory size is kernel configuration option and only one set of kernel configurations
is allowed per project. A project allows multiple builds, but
such builds may only differ by compiler switches, etc. not by kernel configuration
While setting up VxWorks I have maintained two files, Log and TSRs, on
nova both in ~vxworks. Log is a reasonably detailed log of
items relating to vxworks on nova. TSRs is log of TSRs (Technical
Support Requests) submitted to WRS. Some TSRs are still not resolved,
the next administrator may want to follow-up on such TSRs.
On fecode-bd, I have installed standard downloadable kernels for all platforms
given above. My plan was to run these new kernels on my FE
systems, SWIC and vacuum. After the basic correctness of the new
kernels was verified, other FE systems could be upgraded to these standard
5.4 kernels. (It is possible/probable that other systems will
create other issues for the standard kernels, but it seems best to resolve
them one at a time.) However, WRS has changed its PCI support from
VxWorks 5.3.1 to 5.4. This breaks the sld library which both
of my systems use, so I am unable to progress any further until this issues
is resolved. I have contacted WRS about this and was assigned TSR
157622. This TSR has not been resolved yet.
Security, Privacy, Legal