Fermilab Accelerator Control System User Workstation Machine Protection Mechanisms

Kevin Cahill
Updated: 22-Apr-2008


In the past, Fermilab accelerator operator control workstations employed few protection mechanisms for disallowing inadvertent or malicious settings that would adversely affect the performance of the machine. With the new consoles being Linux PCs having far more capability and flexibility than the old PDP-11s, it has been possible to create more sophisticated protection mechanisms. With the expansion in number and geographical placement of new consoles, such measures are a wise investment.


All of the mechanisms described herein are realized in software operating in the consoles and host Linux PCs. They represent a suite of possibilities adequate to protect accelerator operation from a variety of errors. However these mechanisms, being computer software, are in no way fail-safe, and should under no circumstances be thought of as adequate in the utilization of the control system for any function involving personnel protection or safety related compliance.


Twenty PDP-11 computers have served in the past as operator control centers for the accelerators and for these three protection mechanisms exist. The first is a console library routine that returns TRUE if the console is considered privileged. Several applications use this routine to restrict the program's functionality. The second consists of a 32 bit console mask for each addressable device where a 1 shifted by console number is ANDed with the mask providing a logical permission to set the device. This affords a setting capability control on a device and console level. In practice, the maintenance of these masks in the central database is difficult to apply consistently, and each device in the database currently has all consoles enabled for settings. The third mechanism utilizes an application program mask similarly applied to allow applications to be run only on certain consoles.


The issue of affordable protection has been discussed in this division many times. Given past experience, the order of importance of an appropriate mechanism follows:

  1. to protect a console user and the machine from inadvertent mistakes
  2. to protect the machine from repeated inadvertent mistakes
  3. to protect the machine from deliberate but unwanted actions
  4. to provide mechanisms to view settings from all locations

Often these protections are best offered by good application program design. Sometimes discussed is the requirement to gain exclusive control of a portion of the machine impervious to the actions of other users. This level of permission is believed to be very costly in implementation and has a high risk of failure. Our strategy is to balance protection and implementation cost yielding a system that is accessible and accountable.


Several new protection mechanisms are in place.

The console library function LITAPC (LogicalIsThisAPrivilegedConsole) exists and will return TRUE for consoles in the Main Control Room (MCR) and nowhere else. This allows an application program to exclude operations considered inappropriate from non-MCR consoles.

The application mask concept allowing applications to run on a restricted number of nodes has been carried forward. Each new console is a member of one or more console classes. Main Control Room, CDF, Tevatron, and Application Programmer are examples of console classes. The Index Page Editor and protection display functions have changed to reflect these classes. The console index page has been modified to display which programs can be run on each class of consoles. A CDF console for example belonging to the CDF class and Application Programmer class may be able to run CDF programs that are not able to be run by members of the Main Control Room class. Applications that run on the Z index page (debug) inherit the run class bits from the Z program numbers (PA9902-PA9950). This allows the prohibition of execution of programs under development from certain classes of consoles.

The routine LITAPC essentially tests whether the console is a MCR class console. The routine class_do_check will test if a console is a member of a set of classes. The class definitions are available to programmer in an include file. The routine class_get returns the console's class(es). The routine class_get_names will translate class definition bits to ascii. The routine get_setpriv returns the status of the setting lock. Just as LITAPC has been used to provide restrictions from non-MCR consoles, these routines may be used by programmers to provide additional flexibility in protection.

The by-console-by-device mask for data settings has been implemented on the new consoles. The 32 bit mask contained in the database defines the group of console classes allowed to set the device. At the grossest level, CDF and Accelerator Division devices may be separated for control by use of the class mask. On another level, programmer test devices are likely to have a highly accessible class list.

The use of database masks to control setting access are difficult to maintain and do not encompass all user interfaces. We created another class mask for each application that defines the classes of consoles allowed to perform settings from within the application. Programs on the Z index page (debug) inherit their set class bits from the application under test. The user callable routines DPSET, dio_set_xxx, camac_io_c, RPC calls and UPLINK (if imbeddeed in the user application since UPLINK is not a set of library routines, but a protocol of typecoded network messages) use the setting's class bits to prohibit settings from consoles that are not members of the class of consoles not allowed to perform settings. Filesharing writes are restricted from applications that do not have the class set privilege nor the class FSwriteOk definition.

When speaking of consoles and privileges, it is important to remember there are two general categories of consoles. The first is by location. When a console is used in a public way (common username and password), its characteristics for security and privilege are chosen for appropriate access for the console's physical location. The second category of console is by user. Many consoles are located in offices and are started with the office holder's username and password. The characteristics of that console are assigned according to the needs of that individual. Individuals with privileges are always expected to use them appropriately. Consequently, by user consoles are location independent.

A change-program lock exists for each console as well as each user. When the change-program lock is locked, the user may return to the index page but not start any application program. This locks exists for two reasons. After a power outage, most consoles may be locked out by the change program lock until the Main Control Room releases access to the control system. The second use of the lock is to effectively deny setting access to any physical console or individual. When repeated inappropriate settings have been made by an unidentified user at a normally privileged public console or by a normally privileged but unreachable user console, the Main Control Room may wish to `shut down' that console or user. The means to lock out change-programs is presently only available through the console managers.

The Settings lock can be enabled and disabled by the workstation operator allowing DPSET, dio_set_xxx, camac_io_c, RPC and Save/Restore settings to be disallowed. UPLINK is also included if the program has been modified to call class_check_utisets. This affords testing with settings off. Each workstation has a console system manager configurable menu of choices that control the setting lock. The full menu consists of:

Disable 5 Minutes 15 Minutes 1 Hour 8 Hours Forever Remote

A `little privileged' console boots up with settings disabled and only the first menu item (Disable) available to be selected. In order to effect a setting, the user must request that a privileged console user (MCR) unlock his console. A `least privileged' console is a read-only console. Such a console (all consoles remote from Fermilab boundaries for example) may not be unlocked by any mechanism.

A `more privileged' console will boot up with settings disabled and have two menu items available to be selected. That user could unlock his console to allow settings for 5 minutes after which the console would relock. More privileged simply implies more menu items. Only Main Control Room consoles have the full set of items available. Remember that the console still must be a member a class of consoles allowed to set a device.

A `most privileged' console (MCR) will boot up with settings enabled and have the ability to adjust the settings lock remotely on all but the read-only consoles.

`Read-only' consoles present a problem. Many application programs do settings in preparation for a reading. For example, spectrum analyzers must have several controls set to establish the reading's condition. Ignoring this problem would cause potential `read-only' users to justifiably argue for some setting privilege at their console SET4RD, a library routine, supports the encapsulation of setting for reading code. When `set for read' is on, the console's setting lock is ignored. The programmer has the responsibility to encapsulate these settings appropriately. The SET4RD settings are logged at the central, the program's title bar will display `<SET4RD>', and the application's calls to SET4RD are logged in the local console's circular log file and may be viewed by any console's utility window. SET4RD is a privilege defined for each console and may be disallowed for truly `read-only' consoles. Consoles located off Fermilab site property for example do not have the ability to SET4RD.

Let's review the number of ways that a user may typically be denied setting access:

  1. cannot run any program. The change-program lock is locked for this console. It must be unlocked before any programs can be loaded
  2. cannot run the program. This console is not a member of a class allowed to run the selected program. Remember when an application is on the Z index page, its run classes are inherited from the test PA numbers.
  3. program runs, but the application itself refuses to do settings. This application checks and enforces that only certain classes of consoles are allowed to perform settings. These routines are litapc and class_check.
  4. program runs, but library routines refuse to do settings because this console's class(es) do not include the class members allowed to set with this application. The routines so affected are DPSET, dio_set_xxx, camac_io_c, RPC and any UPLINK application that calls class_check_utisets(). Further, Filesharing writes are not permitted if the application does not include the FSwriteOk class defintion.
  5. program runs and tries to do settings but DPSET and dio_set_xxx routines refuse to pass the settings. The database devices attempting to be set are not allowed to be set by any of the console classes this console belongs to.
  6. program runs but library routines refuse to pass the settings because the console's setting's lock is locked. The settings lock is locked and the user is offered no choices for unlocking and the Main Control Room elects not to unlock this user remotely or is not able to unlock the user remotely. The settting's lock affects DPSET, dio_set_xxx, camac_io_c, RPC and any UPLINK application that calls class_check_utisets(). The setting's lock also prohibits the SAVE/RESTORE program from starting a restore from a console whose lock is locked
  7. program runs and tries to do settings encapsulated by SET4RD but is refused. This console is not privileged for SET4RD.

All console lock status changes are recorded in a circular log file viewable by any console's utility application.

As DPSET and dio_set_xxx settings are performed on unlocked consoles, information about the settings are queued and then sent to a central settings logger running on the DCE28 node. Applications using UPLINK, RPC, and SET4RD queue a single `device' setting to the central server of name G:UPLINK, G:RPC, or G:SET4RD. Filesharing writes from applications without class setting privilege but with FSwriteOk class privilege log a single 'device' setting to the central server of name G:FSWRIT. The time, console number, device and property index, and the program name making the setting are logged. For datapool settings, only the first program set of each device is logged to allow for knobbed devices. An application program exists to view the logged settings. Using keys of time, device index, and console location, several presentations provide views of what has happened in the control system. An audit trail following an unfortunate setting would yield such information as console location, application, time of first setting, time console unlocked and relocked, and an application call history preceding and following the setting. Using this information, and the most likely scenario of an inadvertent error, an investigator may be able to identify the responsible individual so the error may be understood and not repeated.

Another application displays logged settings in real time. Again, several views provide immediate settings' feedback to the user.

The ACL command "console_info" displays console privilege information about all consoles.

The index page contains menu selections to aid in the display and control of protection mechanisms. The index page title bar displays the console class(es). Index page entries are displayed in lower case if not runnable, with a white number if not settable by this console's class(es), and with a cyan number when settable. The menu entry `Display Index by Class' displays the index pages as seen by a member of the selected class(es). A MCR operator can view the index page as an SOD class console will display. The menu entry `Display Class Members' will displays the members (console locations and individuals) of the selected class(es). The menu entry `Modify Application Classes' displays the class members allowed to run and set an application and allows suitably privileged consoles to change the classes of consoles allowed to run and set from the selected application. The help facility within the index page details these features.

Security, Privacy, Legal