Frig Front End Emergency Instructions
This page tells you what to do when you need a replacement
PowerPC processor board for a Frig front end. Presumably, you are reading this
because lightning has struck one of the Frig front end CPU cards
or it has otherwise burnt-out or failed, and the primary Frig controls support
engineer is not available.
Before proceeding with the replacement of the CPU card, here are some things
you absolutely should have tried first (unless you see obvious signs of smoke, etc.):
If there is some obvious reason (e.g. smoke), or if you've tried the above
type of things and you are convinced you need to replace the PowerPC VME card,
then you need to follow these steps.
- Reboot the processor. Try it again!
- Telnet into the card. See if it is working at all.
- Check for obvious things like cords unplugged or VME boards being seated properly.
- Take a terminal (e.g. vt220) out to the location, plug it into the
PowerPC's serial console port and see if anything is working.
- Check the network. If things look like they are working on the console terminal,
maybe the network or a network hub/router has failed.
The network hubs for most of the zero buildings are relatively near the Frig VME crate.
See if it appears to be working.
- Check the LEDs on the fronts of the carrier cards for the IPUCD and arcnet cards.
The LED for each slot containing one of the IP cards should be pretty steadily lit up.
If the LED for an arcnet card slot is blinking at 0.1 to 1 Hz, then it is having
problems with the arcnet communications for that node.
When the PowerPC has trouble communicating with only one house of the sector,
it has proven to be a problem with the house's I/O or Thermometry
crates nearly every time.
Here are some things to check.
- Reboot the 186 cards for the I/O and thermometry crates.
- As a desparate measure, reboot the PowerPC after rebooting the 186s.
This usually doesn't help.
- Check the Power supplies for 186 cards for the I/O and thermometry crates.
- Check/swap the 186 cards for the I/O and thermometry crates.
- Only after having done the above (in our experience),
consider replacing the arcnet hardware in the Frig VME crate. Start with the
transceiver module which is mounted on the back of the VME crate.
- Check the VME crate's power supply.
1. Locate a spare PowerPC 2400 processor. You choices are
You always want option A if it is at all possible. The most likely place to
find such a processor is the Frig Test front end in the Linac Annex (first floor,
straight in the door from the Linac hallway, VME crate is on the right).
- A. One already running Frig software, but configured for a different sector.
- B. Other PowerPC 2400 board already configured for VxWorks.
- C. Brand new PowerPC 2400 board. You need to install VxWorks and then
this becomes the same as option B.
If it is a real catastrophe and you need a second spare, the PowerPC for FrigBR
may be available. The frigBR system is generally only a test system,
so you might be able to use it as a second spare. Check with the Cryo coordinator
first. FrigBR is in the B0 service building. Make sure you take BR and not B0.
They are labelled and also B0 has many more arcnet cards connected to it.
If you need more spares than that, you'll have to go on to options B. and C.,
which are slightly more work (i.e. you have to get vxworks installed into the board
for option C.).
2. Assuming you are using option 1.A. and the test frig board, you now
need to change the configuration to that of the target sector (the one you are
The configuration information for any Frig processor is stored in its FLASH
non-volatile memory. This allows for a speedier, network-independent boot.
Recall also that the Frig application code is also bundled with the
VxWorks kernel and stored in ROM.
The configuration information is also in another place: It's in the startup command
scripts which are in the nova directory:
The command file names have the form X.login where X is the sector name:
a, b, c, d, e, f, sy, pbar, and n2400 is the test system's name.
You'll need to run the appropriate sector's command file to get that sector's
Do the following to change the configuration:
- Leave the processor (the one to be reconfigured) running the frig code.
- Login to the processor (the one to be reconfigured) (telnet or console).
- Find out where it's current working directory is:
The most likely answer will be:
- Since we're going to run the command script, we want to start at the
same directory where it the PowerPC starts after a boot. Go to the root:
- Now run the command script for the sector desired.
Note that the last thing in the command script is a command like:
That is the command which starts the Frig application code running.
It is best to temporarily comment out that line for this reconfiguration.
Remember to restore it when you are done.
Run the command script like this:
-> <vxworks_boot/fe/frig/a.login (or b.login, c.login, ...)
There are several key components of this configuration, including setting the
sector name, the number of houses, all the various addresses of the Arcnet and
IPUCD cards, and reading the defaults files. Reading the Frig defaults files
will take a minute or two to complete.
- At the end of that command script, the configuration is in RAM, but not in FLASH.
You must run the following command from the vxworks shell
to write the configuration to FLASH:
The FLASH reconfiguration is now complete. I recommend leaving the IP address and
name unchanged one time and rebooting once. Note that the test node will try
to access arcnets and devices which aren't connected since it is now configured
as the other sector.
- use the VxWorks bootChange command to change the
"inet on ethernet", "gateway inet", and "target name". Also the startup
script must be cleared for the operational sectors. (It is often set to
some value for the test system for testing one thing or another.)
The document frig_boot_params.txt contains
the recent boot parameter settings for each frig node.
- Replace the dead processor with the one you have just configured.
3. If you are forced to use option 1.B. (a PowerPC 2400 not already running Frig),
then life isn't really too complicated.
You need to perform the same basic steps:
- Change the ethernet address, gateway, and target name.
- Use the correct sector's startup script.
- You can actually just set the startup script in the boot parameters
and do nothing more and the PowerPC will function fine as
a frig node. However, recall that the loading of the defaults files from the
startup script (over the network) takes a long time (minutes).
This makes the boot proces uncomfortably slow, but better than nothing.
A better solution is to run the startup script, then execute:-> write_all_defs2flash
Now change the startup script so that instead of calling
it now calls
This makes it read all the defaults out of flash instead of over the network.
- Once you have configured the system, run the startup script one time,
and run write_all_defs2flash, it is really a simple matter to load the
"Frig application plus vxworks" code into ROM.
All you need to do is run
-> flashFile "../2400/vxWorks_rom.bin",870000
(assuming you are in the usual Frig working directory of fe/frig/defaults).
After running write_all_defs2flash and flashFile,
just clear out the "startup script" boot parameter field,
and reboot and everything should work.
See the New Frig Build and Install Instructions
for more details.