European Southern Observatory New Technology Telescope

General :
Observations :
Operations :

Daily Procedures

• to generate the "master calibration" frames from the raw data obtained during the previous night
• to manage and clean the data and product disks on wg5pl

1. Automatic procedures:

    do not run quality maintenance until it is reestablished after upgrade

1.1 Quality Maintenance:

1. Start a session as user quality with a rlogin, and set the display:
pipeline@wg5pl:~ [nn]-> rlogin wg5pl -l quality
password: data-ctrl
quality@wg5pl:~ [nn]-> setenv DISPLAY wg5pl:0.0
2. Start the maintenance procedure:
quality@wg5pl:~ [nn]-> qualityMaintenance
The program will list you all the directories that already contain some processed data, and all the directories that contain raw data. Tell the program which directory you want to work on:
The following product directories already exist:
------------------------------------------------------------------------
1998-06-28   1998-06-30  1998-07-02
1998-06-29   1998-07-01
------------------------------------------------------------------------

Directories available for qualityMaintenance:
------------------------------------------------------------------------
1998-07-01   1998-07-03
1998-07-02
------------------------------------------------------------------------
On which directory should I work?

1998-07-03
If the procedure is ran everyday, there should be only one directory missing from the first list.
3. The program will start an xterm running the RBS (named qm_RBSauto), which will in turn start another xterm (inmidas) running the DRS (that is Midas). In the qualityMaintenance window, the Data Organizer will scan the data directory, and possibly send some Reduction Blocks to the RBS, which should process them in the Midas window. This process may take a while, depending on how much data is present in the data directory.
4. When the whole directory has been scanned, the Data Organizer exits, and you get the following message:
...done.
Wait till the qm_RBSauto and inmidas widows are done too,
then press [Enter]
You have to wait for all the reduction blocks to be processed (it can take a while); when both the windows look quiet, press Enter; you get then:
To terminate, pls type in the RBS window:
 
Stop-DRS
quit
Type the two commands, which must kill the two new windows. This completes the procedure for that data directory.
5. Repeat steps 2 to 4 for the other data directories, as needed.
6. Exit from the quality window.

1.2 Pipeline Maintenance:

pipeline@wg5pl:~ [nn]-> pipelineMaintenance

Please check the log file
  more ~/pm.log

--Error?--v
mv : no match.
--Error?--^

mailx -s "problem with pipelineMaintenance" nttt@eso.org < pm.log

2. Manual procedures:

If you encounter a problem, or if you suspect that the automatic procedures are not working properly, pls report it!

2.1 Quality Maintenance:

• In an xterm, login as quality, and set the display:
rlogin wg5pl -l quality
password: data-ctrl
quality@wg5pl:~ [nn]-> setenv DISPLAY wg5pl:0
• From there, create two new xterm windows
quality@wg5pl:~ [nn]-> xterm -sb -sl 4000 &
quality@wg5pl:~ [nn]-> xterm -sb -sl 4000 &
• >From the quality home directory (first new xterm), start the reduction block scheduler (RBS):
quality@wg5pl:~ [nn]-> RBS
• Start MIDAS through sockets from the RBS window:
RBS_Input: start-DRS
This should bring up another window in which a non interactive MIDAS will run.
• Then set the RBS in automatic mode:
RBS_Input: set auto=yes
• >From the quality home directory (second new xterm), issue the following command:
>createMaster -inst emmi -dir $DHS_DATA/YYYY-MM-DD  -file ~/masterConfig/emmi.mast
where YYYY-MM-DD is the date of the beginning of the last night (yesterday);
The above process will take some time (the more calibrations have been taken over the last 24 hours, the longer it requires). There is no straightforward way for the time being to check that it has been completed. One approach is to monitor the MIDAS via socket (MIDAS55) window. As long as the process is not over, MIDAS commands that are being executed are displayed in this window. If nothing happens in it for a few minutes, the process must have been completed. Then one can check that there are no reduction blocks left in the waiting list by executing the following commands:
>cd $DFS_REDBLOCK
>ls *.rb
>ls *.rW
If no file ending in rb is found and if the only file ending in rW found is set_auto=no.rW, the creation of master calibrations is indeed finished, and you can proceed with the next steps.  If the procedure was not run daily, you may have to repeat this step for all the YYYY-MM-DD directories that have not been processed.
• Once the process is over, from the Unix shell, set the RBS to interactive mode:
>autono
• >From the RBS window, stop MIDAS:
RBS_Input: stop-DRS
• Stop the RBS:
RBS_Input: quit
• Log out from the quality account:
quality@wg5pl:~ [nn]-> exit
and
>exit
(in all the windows where quality is running).
 
• Check for failed reduction blocks and report about them:

pipeline@wg5pl:~ [nn]-> cd $DFS_LOG
pipeline@wg5pl:~/DfsLog [nn]-> ls |grep bad
If in the list, there is a file with the name RBS_bad.YYYY-MM-DD for the date of the beginning of the night, e-mail it to Michele Peron:
pipeline@wg5pl:~/DfsLog [nn]->
mailx -s "Failed reduction blocks" mperon@eso.org, nttt@eso.org <RBS_bad.YYYY-MM-DD
 
  
• Next steps are aimed at maintaining disk space.

Manage the logs: stay in $DFS_LOG and do the following, for all the dates except the current one and the previous 2 nights:
pipeline@wg5pl:~/DfsLog [nn]-> rm RBS_bad.YYYY-MM-DD
pipeline@wg5pl:~/DfsLog [nn]-> rm RBS_old.YYYY-MM-DD
pipeline@wg5pl:~/DfsLog [nn]-> rm RBS.YYYY-MM-DD
pipeline@wg5pl:~/DfsLog [nn]-> rm DO.YYYY-MM-DD
pipeline@wg5pl:~/DfsLog [nn]-> cd $DHS_LOG
pipeline@wg5pl:/data/ArchLog [nn]-> rm DOSubscribe.YYYY-MM_DD
• Manage the pipeline outcome:

pipeline@wg5pl:~ [nn]-> cd $DFS_PRODUCT
The command ls should show you a list of directories with names of the form YYYY-MM-DD. Keep only the last 3 directories (for the current date and for the previous 2 dates). Delete the older directories and their contents:
pipeline@wg5pl:~ [nn]-> rm -rf YYYY-MM-DD/
Note: if the present procedure is duly executed every day, there should never be more than one directory to delete.
• Manage the reduction blocks:

pipeline@wg5pl:~/DfsProduct [nn]-> cd $DFS_REDBLOCK
pipeline@wg5pl:~/DfsRedblock [nn]-> mkdir YYYY-MM-DD
pipeline@wg5pl:~/DfsRedblock [nn]-> mv O*.rX YYYY-MM-DD/
pipeline@wg5pl:~/DfsRedblock [nn]-> mv O*.rY YYYY-MM-DD/
pipeline@wg5pl:~/DfsRedblock [nn]-> mv O*.rZ YYYY-MM-DD/
BEWARE !!: Do NOT move the reduction blocks with name *.rW !!!
YYYY-MM-DD is the date of the beginning of the last night. Once this is done, keep only the directories pertaining to the previous 3 nights; remove any older directory.
• Manage the raw data:

remove all the directories but last three ones:
pipeline@wg5pl:~/DfsRedblock [nn]-> cd $DHS_DATA
pipeline@wg5pl:/data/ArchData [nn]-> rm -r YYYY-MM-DD/

Send comments to: NTT Webmasters <nttweb@www.eso.org> Last Update: MMM DD, YYYY (WHO)