Intructions for Starting a Run

or for Recovering from a Crash

(Version of 25 Oct 2009)

Crash Recovery

If our MIDAS web server is not running, start it by typing:

pc7912]$ mhttpd -D

If that doesn't work, try clearing the shared memory files and segments:
pc7912]$ penipcrm
Then try again.

Now load the recovery odb:
pc7912]$ odbedit -c "load /data/recovery.odb"
Then proceed as follows.


If the MIDAS web server was already running, you can ignore the steps above.

To start the DAQ system after a crash it is a good idea to stop all frontends, clear the shared memory segments, and load the recovery odb, before trying to re-start the component that crashed.

To do this you may use the custom Controls page.
If the page is inaccesible, that means our MIDAS web server is not running and our odb is not loaded. See above.


• Click the "Stop Clients" button.
  This may take a few minutes to complete. This will stop ALL the MIDAS clients, including mhttpd (so the MIDAS status page and the "Controls" page will become inaccessible for a moment. Once all clients have been shut down it will remove the shared memory files and segments, then start mhttpd again automatically, and you will have access to these web pages again. If you are not continually pressing "refresh" you may not even notice that the page was down.)
  
  
  
• Now start the frontends.
  Click the individual buttons to start each frontend individually. Start them in the proper order: Slow Control, 0, 1, 2, 3. But don't start the next one until the previous one has initialized.
  
  Some frontends take some time to initialize. You should have another browser window open to the MIDAS Status page. Clicking refresh, will update the messages. After clicking a button to start a frontend, wait for a message to appear that says that frontend equipment has been initialized, e.g.:
  [Trigger Frontend01,INFO] Program Trigger Frontend01 on host pc809 started
  [Trigger Frontend01,INFO] Frontend01 equipment initialized
  You will have to refresh the MIDAS Status page often.
  
  Alternatively, you could run odbedit in a terminal, and you will see all the messages.
  
  
  
• Click the "Start mevb" button.
  This starts the event builder running on the backend.
  
  
  
• Click the "Start logger" button.
  This starts the logger on the backend.
  
  
  
• Click the "Start analyzer" button.
  This starts the analyzer on the backend.
  
  
  
• Now you should be ready to start a new run.
  Check that everything is green on the MIDAS Status page. If so, start a new run. If something is still red, try the above steps again or do it manually as described below.
  
  

  ---

  
  
  Instructions for Manually Starting our Data Aquisition Programs
  
  
  You will most likely start all the programs from the Backend computer (pc7912), so you might want to open at least six terminal windows.
  
  Note that you should start the frontends in the proper order: fe0 then fe1 then fe2 then fe3.
  
  
  
• In the first terminal start the backend daemons:
  
  You should always check to see if the daemons are already running before you start them so you don't have multiple instances of them.
  
  If you suspect they are causing problems, you can kill them, then restart them. You can kill them by executing the following command MULTIPLE times for each process (mhttpd, mevb, mlogger, analyzer). Execute the command until you see that no process has been killed, for example,
  
  pc7912]$ killall mlogger
  pc7912]$ killall mlogger
  mlogger: no process killed
  
  Then:
  
  pc7912 ~]$ odbedit -c "load /data/recovery.odb"
  pc7912 ~]$ mhttpd -D
  pc7912 ~]$ mevb -D
  pc7912 ~]$ mlogger -D
  pc7912 ~]$ ~/online/penanalyzer2009/analyzer -D
  



• Now start mozilla/firefox and go to:
  
  http://pc7912 to see our status page.
  
  
  
• In the second terminal start Frontend00:
  
  pc7912 ~]$ ssh -Y pc6166
             enter canonical pen password (only if asked!)
  pc6166 ~]$ killall fe0
  pc6166 ~]$ killall fe0
  pc6166 ~]$ killall fe0
  pc6166 ~]$ cd ~/online/fe0
  pc6166 fe0]$ ./fe0 -i 0 -h pc7912
  
  The key "-i 0" tells it that it is the #0 frontend.
  The key "-h pc7912" tells it the host (or Backend) where MIDAS is running.
  



• In the third terminal start Frontend01:
  
  pc7912 ~]$ ssh -Y pc809
             enter canonical pen password
  pc809 ~]$ killall fe1
  pc809 ~]$ killall fe1
  pc809 ~]$ killall fe1
  pc809 ~]$ cd ~/online/fe1
  pc809 fe1]$ ./fe1 -i 1 -h pc7912
  



• In the fourth terminal start Frontend02:
  
  pc7912 ~]$ ssh -Y sluser@pcp035120
             enter canonical pen password (while the user here is "sluser")
  pcp035120 ~]$ killall fe2
  pcp035120 ~]$ killall fe2
  pcp035120 ~]$ killall fe2
  pcp035120 ~]$ cd ~/online/fe2
  pcp035120 fe2]$ ./fe2 -i 2 -h pc7912
  



• In the fifth terminal start Frontend03:
  
  pc7912 ~]$ ssh -Y pc7196
             enter canonical pen password
  pc7196 ~]$ killall fe3
  pc7196 ~]$ killall fe3
  pc7196 ~]$ killall fe3
  pc7196 ~]$ cd ~/online/fe3
  pc7196 fe1]$ ./fe3 -i 3 -h pc7912
  



• In the sixth terminal start the Slow Control Frontend:
  
  pc7912 ~]$ ssh -Y pc4821
  pc4821 ~]$ killall sc_fe
  pc4821 ~]$ killall sc_fe
  pc4821 ~]$ killall sc_fe
  pc4821 ~]$ cd online/sc_fe
  pc4821 sc_fe]$ ./sc_fe -h pc7912
  



Start the run.





---



Assuming all pc's are up and running, and the drivers are installed in the frontends, you may follow the procedure listed above.

If you reboot FE0 or FE1 or FE2, you must re-install the drivers for the CAMAC interface (FE0), VME interface (FE1) and Acqiris Digitizer (FE2)


A. Palladino 18 Oct 2009