EPICURE Design Note 6.3<P> <b> Use of the QVI and Common VME Memory</b>

EPICURE Design Note 6.3

Use of the QVI and Common VME Memory

F. J. Nagy and A. D. Thomas

Introduction

The EPICURE system acquires data using VME-based Data Acquisition Engines (DAE) connected to MicroVAX hosts. The QBus to VME Interface (QVI) permits the MicroVAX to access locations in the VME address space. Communication between the MicroVAX and the DAE processors is via the Common Memory (CM) area in a VME memory module. All communications between the MicroVAX and the DAE and between the various processor systems of the DAE are done by passing messages using queues within the CM.

The organization of the Common Memory is described and the fixed, global area offsets at the base of Common Memory are detailed in this document. The C header file in the area contains the definitions of the structures and constants described here. This file also contains definitions particular to various processors enclosed in ``#ifdef'' blocks using the preprocessor symbols (defined by the VAX/VMS C compiler) and (defined by the Green Hills 68020 compiler). The 80386 definitions are the same as those used for the VAX.

Common Memory Block Format

This description of the format of a CM block (CMB) is from EPICURE Design Note 3.5, VMS System Services for QVI Support. The CM blocks contain the messages passed between the host and the processors of the DAE using the CM queues. CMBs are fixed length memory blocks; several free lists for the various sizes of the CMBs are maintained in the CM. The higher-level QVI services automatically select a CMB size based on the length of the User Memory Block (UMB) provided by the caller. The format of a CM block is:

A detailed description of the CM block structures can be found in EPICURE Design Note 3. These data structures are defined in as the struct's , (a prototype UMB) and (again using the prototype UMB). For purposes of our discussion, there are only two items of interest in these data structures, the CYCLE bit in the flags word and the ``Return Queue Handle''.

The definition is that of a ``prototype'' UMB with 8 bytes of ``data''; this definition is for purposes of laying out the UMB header. It is expected that users of the the QVI services will generate their own UMB definitions starting from this ``prototype'' by replacing the ``data'' definitions with their own actual data fields.

We recall from Design Note 3 that the Return Queue handle is used by a VME processor to return a message to the VAX by specifying one of several possible queues. The CYCLE bit is used to indicate that the queue entry be placed back onto an externally defined default DAE queue (always assumed to be the Timer queue at this time) after processing has been completed by the QVI services on the VAX. The cycle was begun by the VAX placing a message on a request queue for one of the processors in the VME crate. This is a simplified but essential description of the ``outer protocol'' for passing messages from the VAX to micros and back.

Data Acquisition List Format

A Data Acquisition List (DAL) for the DAE can be the block contents of a UMB which has its block identifier set to identify it as a DAL. The format of a DAL is:

The format of a Data Acquisition Packet (DAP) in the Data Acquisition list is:

The target node in the initial configuration of EPICURE systems is simply the one and only VME crate connected to a VAX front-end computer. In the future it could be the node number of a network of VME crates emanating from the VAX front-end. A node number of zero always refers to the local crate (local to the attached VAX or to the processor in the VME crate).

The Queue Table Index or QTI, is used as a short form of the target queue handle. It directly indexes a table of queue header addresses (QVI routines are provided to turn a QTI into a queue handle). It specifies which module in the crate is to process the current DAP. The length field is an unsigned 16-bit integer and specifies the length of the DAP (including node #, QTI and the length field itself) in bytes. The node number and the QTI lumped together are sometimes referred to as a Processing Element Address (or PEA).

The second and third longwords of the DAP contain the length of the data transfer and the offset for array-type devices. The DAP comes from the EPICURE database with default values for these fields; the Data Acquisition Requestor (DAR) process will overwrite these values if the user specifies non-default offset or data length. For write (SET) operations, the data to be written is attached to the end of the DAP and the length of the DAP incremented. The data length field specifies the amount of data added for the write operation. The location of the write data in the DAP is specified by the ``DAP offset to SET data'' field which contains the offset, relative to the start of the DAP, to the write data. This value is a constant stored in the DAP as gotten from the database; the write data offset is ignored for read operations as no write data is attached to the DAP in such cases.

Example usage of DAL and DAP Structures

The following pseudocode describes the sequence for handling a single VAX request. There may be multiple such requests in various stages of execution at any instant.

On the Front-end MicroVAX:

DAS builds DAL
DAS calls to allocate a common memory block, copy the new DAL to it and insert the block onto the default DAE queue (the default queue is owned by the timer module)
QVI service interrupts DAE default processor

On the VME side (initially the Timer processor):

Receive (request queue not empty)
DOWHILE request queue NOT empty
ENDWHILE

Data Acquisition List Expunger

In addition to DALs, the DAE processors must also handle messages designed to ``expunge'' outstanding DAL requests. When the Data Acquisition Server (DAS) process receives a request to cancel an outstanding DAL it will first use the QVI services to clear the CYCLE bit in the Common Memory block containing the DAL. If the DAL then completes, it will be deleted automatically by the QVI services. However, the DAL may be in a wait state and held by one of the DAE processors. Therefore, DAS will build a DAL expunger message and invoke the QVI Multicast services to send this through all the DAE elements.

Common Memory Organization

The VME Common Memory is organized into multiple areas. The major areas are the Global Area, the Queue Headers and the Allocation Areas. The organization of the CM is defined by the Common Memory initializer program which is responsible for initializing the CM and setting up all the initial data structures in the CM. Once initialized, the CM itself contains all necessary descriptive information. This setup permits the organization of the CM to be modified by changing just the Common Memory initializer; other software using the CM will automatically adjust at run-time to the change in the CM organization.

The Global Area starts at the base of the Common Memory. Items within the Global Area mostly consist of longwords which have a fixed offset from the base of the CM (see the detailed description of the Global Area below). Primarily the Global Area consists of scalar items and CM-relative pointers ( offsets relative to the CM base, also called CM-pointers in this document) to other areas and structures in CM. In addition, the CM Global Area contains the Data Acquisition Protocol (DAP) queue table. This table establishes the relationship between the DAP queue index and the CM queues.

The Queue Headers is an array of the headers for the CM queues. This array is in a contiguous area pointed to by a location in the Global Area. The extent of the array is maintained in the Global Area as a count of the queue headers in the array.

The Allocation Areas contain the variously sized Common Memory Blocks and constitute the bulk of the CM. Each Allocation Area is described by an Allocation Area Descriptor (AAD). The AAD points to the start of its associated Allocation Area, specifies the size of the individual blocks in the Allocation Area and the count of the blocks in the Allocation Area. The AAD also contains a CM-pointer to the queue header which maintains the free list for that Allocation Area. The Global Area contains a CM-pointer to the contiguous array of AADs and a count of the number of AADs in the array. In addition, a pair of CM-pointers in the Global Area point to the bottom and top of the Allocation Areas to provide a means of validating CMB pointers; this assumes that the Allocation Areas are setup in a contiguous section of Common Memory.

DataPools are ( moderately) large blocks of Common Memory set aside as data storage areas. DataPools were added for the SWIC Datapooling Services which use a VAX-based process to collect the SWIC data using the CAMAC 032 modules and deposity this data into Common Memory (in a DataPool). When users request SWIC data, their data acquisition requests were directed to extract the information from the SWIC DataPool (by the DAE CAMAC processor). The DataPools are normally allocated starting from the top of Common Memory.

Detailed Description of Global Area

The Global Area described as a C structure (in ) is:
#define CMG_C_MAXDAPQS  256
#define	CMG_C_MAXPOOLS	4
       
struct CMGlobals {
    int cmg_l_chkini;
    unsigned long *cmg_a_cmvme;
    unsigned long cmg_l_cmsize;
    unsigned short *cmg_a_eirvme;
    unsigned long cmg_l_cqhoff;
    unsigned long cmg_l_cqhcnt;
    unsigned long cmg_l_aadoff;
    unsigned long cmg_l_aadcnt;
    unsigned long cmg_l_baabot;
    unsigned long cmg_l_baatop;
    unsigned long cmg_al_dapqtbl[CMG_C_MAXDAPQS];
    unsigned long cmg_l_clink;
    unsigned short cmg_aw_tclk[16];
    unsigned long cmg_l_qdboff;
    unsigned long cmg_l_qdbcnt;
    unsigned long cmg_l_muxoff;
    struct DPD cmg_ar_dpds[CMG_C_MAXPOOLS];
    };
list [cmg_l_chkini] contains the value to indicate that the Common Memory has been initialized. [cmg_a_cmvme] contains the VME address of the Common Memory module (this is redundant information since both the VAX and the DAE processors must have prior knowledge of the VME address of the CM). [cmg_l_cmsize] contains the size, in bytes, of the Common Memory. [cmg_a_eirvme] contains the VME address of the External Interrupt Register on the QVI board for use by the DAE processors. [cmg_l_cqhoff] contains the CM-pointer to the start of the array of CM queue headers (these form a contiguous entity in the CM). [cmg_l_cqhcnt] contains the number of CM queue headers in the array pointed to by cmg_l_cqhoff. [cmg_l_aadoff] the CM-pointer to the start of the array of Allocation Area Descriptors (these form a contiguous entity in the CM). [cmg_l_aadcnt] contains the number of AADs in the array pointed to by cmg_l_aadoff. [cmg_l_baabot] contains the CM-pointer to the start of the CM block Allocation Areas. [cmg_l_baatop] contains the CM-pointer to the top (first byte past the end) of the CM Allocation Areas. By assumption, the CM Allocation Areas for the various sized CMBs form a contiguous entity in the CM; this quantity and cmg_l_baabot are used to validate pointers to CMBs. [cmg_al_dapqtbl] is the DAP queue table which is indexed by the DAP QTI (an unsigned byte). The contents of a table entry is a CM-pointer to the queue header associated with the queue index (0 indicates no queue for an index). [cmg_l_clink] contains the time of the year in clinks (number of seconds since midnight of January 1, 1972). This cell is initialized by the VAX at boot time and re-initialized at regular intervals. It is updated at one second intervals by the Timer DAE processor. [cmg_aw_tclk] is the array of 256 bits for Tevatron Clock (TCLK) events. This array is initially clear by the VAX and updated by the Timer processor. [cmg_l_qdboff] the CM-pointer to the start of the array of Queue Demultiplexing Blocks. The QDBs form a contiguous entity in the CM. [cmg_l_qdbcnt] contains the number of QDBs allocated. [cmg_l_muxoff] the CM-pointer to the queue header of the multiplexed VAX return queue. [cmg_ar_dpds] array of DataPool Descriptors giving for the optional data pool areas.

Queue Header Format

A CM queue is a singly-linked FIFO list. The head pointer is the CM-pointer to the first entry on the queue (or 0 if the queue is empty). The first longword of each queue entry contains the CM-pointer of the next entry; the last entry contains a 0 in the link longword (see Section 1.1, Common Memory Block Format). The tail pointer is the CM-pointer to the last (tail) entry on the queue (or 0 if the queue is empty).

A CM queue header occupies 8 longword (32 bytes). The format of a CM queue header is:

A detailed description of the CQH (Common Memory Queue Header) structure can be found in EPICURE Design Note 3.

Description of Allocation Area Descriptor

Each Allocation Area of fixed-size Common Memory blocks is described by an AAD. The AAD is setup by the Common Memory initializer program and is used by the Common Memory Dump Analyzer program and by the CM interface packages during their initialization. For instance, the VAX QVI services scan the array of AADs to construct a sorted table describing the CM free lists. This table provides for rapid lookup of the address of the CM queue header for the free list with blocks of a particular size. The format of the AAD is:

The AAD structure for C is:

struct AAD {
    unsigned long aad_l_offset;
    unsigned short aad_w_size;
    unsigned short aad_w_count;
    unsigned long aad_l_fqhoff;
    unsigned long aad_l_spare0;
    };

Description of DataPool Descriptor

Each DataPool in Common Memory is described by a DPD structure located in an array of DPDs in the Global Area of Common Memory. The DPD is setup by the Common Memory initializer program in response to DATAPOOL commands which allocates the space for the DataPool and records the offset to the base of the DataPool and its size in the DPD. The QVI and CM interface packages use the DPD to return the DataPool location to their users. The internal structure of the DataPool is up to the cooperating processes; it will usually be an array of structures.

The DPD also includes a longword used by the DataPool locking services in these packages. At this time, only the lower byte of this longword is actually used, but the remainder of the longword is reserved to allow the locking algorithms to be changed to a mutex to allow multiple simultaneous DataPool readers.

The format of the DPD is:

The DPD structure for C is:

struct DPD {
    unsigned long offset;
    unsigned long size;
    unsigned long lock;
    };

Description of Queue Demultiplexing Blocks

The QDB is stored in the Common Memory but only used by the VAX. The QDB describes the return queue and notification AST address for those VAX processes which use the multiplexed return queue. The process services interrupts on from the multiplexed return queue and uses the originator identifiers to put blocks on the appropriate return queues. The format of the QDB is:

The QDB structure for C is:

struct QDB {
    unsigned long qdb_l_pid;
    unsigned long qdb_l_astadr;
    unsigned long qdb_l_rtnqoff;
    unsigned char qdb_b_flags;
    };

#define QDB_M_EINXP 1

The EINXP flag is used by the process. If the destination process does not exist, the CM blocks are released to the free list rather than being placed on the return queue for the non-existant destination process.

Keywords: EPICURE, VME, QVI, Common_Memory.

Distribution:

Security, Privacy, Legal

rwest@fsus04.fnal.gov