Alarm Report Distributor
December 14, 1994
This document describes the implementation of the Alarm Report Distributor (ARD). Readers needing a description of the functionality of an ARD should see EPICURE design note 87 ``An Overview of the Epicure Alarm Subsystem'' (sections 6, 7 and 8) and design note 96 ``User Interface to ARD''. In particular, the reader should be familiar with the distinction between a master and a secondary ARD. Unless the context implies otherwise, the term ``alarm'' will be understood to include events.
The collective purpose of the ARDs is to receive alarms from AMP processes running on one or more front end nodes and to distribute them to display processes that may be running on any node . To accomplish this the master ARDs must maintain links with an AMP, a DBserver, the other master ARDs, and any number of secondary ARDs and display processes. The secondary ARDs must maintain links with only one master ARD, a DBserver , and any number of display processes.
The link to an AMP is maintained using a permanent mailbox which is created and continually monitored by the ARD. The AMP passes raw alarm reports to the ARD through the mailbox. The link to the DBserver is via the DB_ routines in the epicurelib library. Links to other master ARDs and to secondary ARDs are via DECnet (assisted by the SNS routines). Communication from a display process to an ARD is via the same mailbox that is used by the AMP. Communication from the ARD to a display process is via a temporary mailbox created by the display process. All communication, except to the DBserver, is asynchronous.
The ARD is divided into eleven modules. The modules are grouped into three categories: the central modules that provide the functionality of the ARD, the interface modules that provide the communication interfaces (to the AMP, the other ARDs, and the display processes), and the utility modules that perform some service (such as encapsulating a data structure or maintaining a log file). Some of these modules will be partially or entirely implemented as ASTs.
Messages are passed between the various modules of the ARD. A message may be a raw report, a cooked report or a request. Raw and cooked report messages contain raw and cooked alarms respectively. Request messages contain requests from display processes (such as connect requests or requests to set a new filter), but also may also contain status messages. An example of a status message would be an announcement that the AMP on a particular node has crashed.
The central modules:
The interface modules:
The utility modules:
The files epicure_sys_inc:alarmnet.h and epicure_inc:alarms.h define the structures used to contain messages.
Messages that are sent among the other ARDs, the AMP, and the display processes will be sent in packets. A packet will be preceded by a header containing the following data: a length field containing the length of the entire packet; a count field containing the number of messages in the packet; and a type field indicating whether the packet contains raw reports, cooked reports, or requests. Note that while the messages in a particular packet must be of the same type, they may have different lengths.
Messages that are being passed around inside an ARD are contained in a structure that is an overlay (union) of the structures that define requests, raw alarm reports and cooked alarm reports.
The unpacker disassembles packets of messages. The unpacker is invoked by the network receiver and the mailbox receiver (at AST level) when they receive a packet of messages. The individual messages are passed to the cooker or to the cooked report handler to be queued for later processing by the scheduler in the main loop. A sys$wake call will be made to wake up the scheduler, if it is hibernating.
The cooked report handler maintains a queue of pending cooked reports and requests. The unpacker and the cooker use a utility routine provided by this module to enqueue cooked reports as they arrive or are cooked. This routine also makes a sys$wake call to notify the scheduler that a message is waiting to be processed.
When the scheduler invokes the cooked report handler, the cooked report handler removes one item from the queue, determines the type of message it contains, and passes it to one or more of the network receiver, the display manager and the current list manager.
The cooker maintains a queue of pending raw reports. The unpacker uses a utility routine provided by this module to enqueue any raw reports that it receives. The cooker uses the DB server to cook the raw reports. The cooker also makes a sys$wake call to notify the scheduler that a message is waiting to be cooked.
When the scheduler invokes the cooker, the cooker will remove one raw report from the queue, cook it, and pass it to the cooked report handler.
The scheduler invokes the cooker and the cooked report handler in that order of priority. Both of these modules return a value indicating success if there was any data in their respective queues for them to process, otherwise it returns a value indicating failure. If both of these modules fail (i.e., if their respective queues are empty), then the scheduler hibernates. The utility routines that enqueue messages issue a sys$wake call whenever they enqueue a request or report. The scheduler is also the entry point to ARD, and does the initialization.
The mailbox receiver maintains a permanent mailbox (``RDCS$ARD_MAILBOX'') into which the AMP and the display processes place packets of raw reports and display requests, respectively. When a packet arrives in the mailbox, a read completion AST is invoked which removes the packet from the mailbox, and passes it to the unpacker.
The network receiver sets up and maintains links to the other ARDs. A logical (``RDCS$ARD_-MASTER_NODES'') is defined as a list of all the nodes running master ARDs. If the name of the local node is on this list, then the ARD is a master ARD, otherwise it is a secondary ARD. If the ARD is a master ARD, it uses the SNS routines to enable incoming connect requests. After the incoming connections have been enabled, it tries to establish a (transparent) link to all of the nodes other than itself on the list of master ARDs. Only one link is allowed between any pair of nodes, therefore any incoming connect request that would duplicate a link is refused. If an established link goes down, a master ARD assumes that the other process has gone down and will re-establish the connection when it comes back up. For each link established with a master ARD, the master ARD sends a dump of the alarms in the list maintained by the current list manager. If the ARD on the other end of the link is a master ARD, then only the alarms that originated in the AMP on the local node are sent; if the other ARD is a secondary ARD, then all of the alarms are sent.
If the ARD is a secondary ARD, then it does not use the SNS routines, but tries to establish a transparent link to the first ARD on the list of master ARDs defined by the logical. If the link between a secondary ARD and a master ARD cannot be established (or if an established link fails), then the secondary ARD will attempt to fail over to one of the other master ARDs on the list. If no connection can be established to any master ARD then the secondary ARD goes into a polling loop where it periodically tries to make a connection.
In both master and secondary ARDs, if a link to a master ARD goes down, then a status message is generated stating that the ARD on the other end of the link is no longer reachable. The ID in this message is set as though the message had come from the ARD that went down. The message is passed to the cooked report handler for distribution.
All incoming message processing is done by read completion ASTs; note that two different ASTs must be provided since SNS links have a different protocol from transparent links. Packets arriving over a link are passed to the unpacker. The unpacker will pass the messages in the packet to the cooked report handler (No raw reports are expected to arrive over the network).
The cooked report handler sends cooked reports and status messages to the network receiver. If the ID of a message (or report) matches the ID of the local node , then the message is sent out over each link. If the ID of the message does not match the local node , then it is sent over only those links that lead to a secondary ARD (each master ARD is directly connected to every other master ARD, so it is assumed that the sender of the message sent it to all). Each link has a queue to hold messages waiting to be sent; a timer routine periodically checks each queue and sends out (in packets) any waiting messages.
The display manager is responsible for distributing cooked reports to the display processes and for handling requests from the display processes. Communication from the display manager to a display process is through a temporary mailbox created by the display process. Communication from the display process to the display manager is through the permanent mailbox created by the ARD (via the mailbox receiver and cooked report handler). A display database is maintained that contains the following data for each display process: The unit number of the mailbox created by the process (this serves as the identifier of the process), the process id number of the display process, the filters currently in use by this process, a pause flag, and an output queue containing messages waiting to be put into the mailbox. The initial setting of the filter allows all messages to be sent to the display process; the initial value of the pause flag is ``pause enabled''. Messages to be put into a particular mailbox are placed in the corresponding output queue; a timer routine periodically checks each queue and sends out (in packets) any waiting messages (similarly to the network receiver).
If the display manager finds that a display process has died (this will be assumed if the display process takes too long to empty its mailbox, or if the queue gets too long), then the process will be disconnected as though a disconnect request had been received. The display processes will be expected to declare an image rundown AST that will serve to notify the process if the ARD goes down unexpectedly. The address of this AST will be included in the connect message.
The display manager responds to the following messages:
The current list manager maintains a list (or some other data structure) of all outstanding alarms. If a going-bad alarm is received it is added to the list (if it is already on the list, then the old one is replaced). When a going-good alarm is received the corresponding going-bad alarm is removed from the list (if it is there; note also that the going-bad alarm must be dated earlier than the going-good alarm for it to be considered ``corresponding''). If an event is received, and the hold bit is set, then it is added to the list. It will remain on the list until it is cleared by the AMP.
A separate list is maintained for status messages. On this list will be a status message for each ARD and AMP in the alarm system. Initially all other ARDs and the AMPs are assumed down; the list is updated as links are established. ARD status messages are generated in the master ARDs and are sent only to secondary ARDs and display processes (since all master ARDs are mutually connected, they are all aware of each other's status). AMP up status messages are sent by the AMP (or the UTI). When an AMP up message is received from the AMP, the current list manager sends a request to the mailbox receiver to set up a polling loop to monitor the AMP. This polling loop generates an AMP down message if the AMP process is found to have gone away.
If a request for a dump is received (from the network receiver or the display manager) then all of the outstanding alarms and status messages are copied into a (new) list and returned. If the request is for a local dump, then a list of only the outstanding alarms that originated on the local node is returned. AMP status messages will always be first in the list, followed by ARD status messages, followed by alarms and events. Note: in order to prevent spurious status messages from being sent when the ARD is starting, all dump requests received are queued until the ARD has been up for a predefined amount of time (a few seconds). After this time, dump requests are serviced as they are received.
The logger maintains a log file in which significant events are recorded. Examples of significant events are status messages received, network links made or lost, error returns from the DBserver, etc. An interface is provided so that other modules can request that messages be printed. Each message is preceded by a time stamp.
In order to avoid continuous disk activity and to allow inspection of the log file while the ARD is running, the log file is only accessed at intervals (typically 15 seconds). All incoming messages are queued and are only actually written at these intervals. Every night, at midnight, the logger prints a list of the status messages, and a status dump of every virtual memory zone used by the program . Also, in order to prevent the log file from becoming too large, a new version is created at this time. The old version is mailed to the list of users specified by the logical ``RDCS$ARD_MAILING_LIST.''
The memory manager handles the interface to the LIB$ memory allocation routines. In order to prevent excessive copying of messages as they are passed from one module to several others, reference counting is used. This is also managed by the memory manager. The disadvantage to this is that each module must accurately tell the memory manager when it is through using a message so that it can be freed.
The memory manager also contains a set of queue management routines that are used by other modules.
The misc module contains assorted routines that don't fit anywhere else. Mostly these are interfaces to LIB$ and SYS$ library routines.
The following executables are kept in the directory EPICURE_ROOT:[WORK.ARD.DEVOY.RELEASE_3]:
Keywords: Epicure, program, alarm, event, ARD, almdisp, AMP