|
European
Southern Observatory
New Technology Telescope |
| General : | |||||
| Observations : | |||||
| Operations : |
![]()
Procedure:Data Flow Problems
Files are not transferred from machine to machine. Author:Olivier R. Hainaut Releases:3.0/2001-jul-17: updated for Jun.01 upgrde
2.1/2001-jan-3 update for feb00 upgradee
2.0/1999-jul-5: updated for Jun99 upgrade
1.1/1999-Jan-11 : creationIf you suspect that the Data Flow System (DFS) is not transferring properly the file from machine to machine, apply the procedure given in Section 1.
Section 2 (Operation) lists the commands and info to start and stop the processes.
This procedure deals only with the "Back End" of the DFS (i.e. the one that transfers the files from machine to machine). The front end (i.e. P2PP/OT) is dealt with in another recovery procedure.
1. Recovery Procedure
Follow this procedure point by point; do the jumps in the procedure as described.
Two cases are considered:
- a Quick Recovery scenario, that should be OK in most of the cases
- a Deep Recovery scenario, in case the Quick recovery does not work. You are potentially in trouble, and may want to get lsArchive (Anton/Saul) or SWC psychologically prepared to help.
Sections in italic are for Deep Recovery, potentially with the assistance of lsArchive or SWC; hwv, the procedure below contains enough information so that even a visiting astronomer should be able to solve the problems.
To do the test described, log on the machine considered using the normal operation account (tcs, cam, sofi, ins, pipeline, astro, archeso); they can all log into each other via rlogin without password (except astro: anybody can log as astro, but astro cannot log in as someone else).
- 1.1 Are the images created on wemmi/wsofi
- cd $INS_ROOT/SYSTEM/DETDATA
- ls -lrt: is the last image what you expect?
- yes: continue below
- No: Do you have enough disk space?
- yes: your problem is not a Data Flow System problem. Quit this procedure and investigate the problem.
- No: clean the disk and retry!!!
- 1.2 Is vcsolac running?
- check vcsolac on wemmi/wsofi, on nttexecDfsCheckAll
- Green?
- Quick recovery: goto 2.DHS
- Deep recovery: continue this section
- Red? continue this section
- psg vcsolac : do you get a process?
- yes, eg: vcsolac -dhshost wg5dhs .....: continue
- no: restart vcsolac, cf Normal Operation, and retry
- Is vcsolac running properly?
- cd $DHS_LOG
- more (or tail) VCSOLAC-wemmi-ONTT.2001-07-17 (change as appropriate). For each image, you should get "...processing /vlt/...RILDOT01.12.fits" OK?
- yes: vcsolac is running properly
- No: restart vcsolac and retry
- Is it running properly now?
- yes: have a good night
- no: you are in trouble; you will have to transfer the data via FTP from wemmi/wsofi, and report that vcsolac is broken.
This machine is under lsArchive's responsibility. You can do some diagnostic here, but no intervention. This procedure may be of help in case nobody of lsArchive is reachable. Note that starting/stopping processes on this machine is likely to screw up the Archive, so beware and let lsArchive know that you did something.
- 2.1-Are the images transferred on wg5dhs?
- cd /data/raw/2001-02-03
- ls : is the last image what you expect (the filenamess are in the ONTT.2001-... format) ?
- yes: wg5dhs is OK: go to next machine
- No. Do you have enough disk space?
- No: clean the disk and retry!!!
- Yes: Perform step "1.2- is vcsolac running?"; still does not work? you are in trouble. Call lsArchive/SWC; the remainder of this section is in case SWC cannot help (e.g. no SWC), but should be used only in case of emergency.
- 2.2-Is the DHS running?
- check the dhs light on wg5dhs on nttexecDfsCheckAll
- Green?
- Quick Recovery: goto 3.
- Deep recovery: continue this section
- Red: continue this section
- psg dhs : it must return --among other processes-- dhs -dhsdata /data/raw -logpath ......
- yes. Check the Log file:
- cd /data/msg
- more (or tail) DHS-wg5dhs.2001-02-03; you should see "...received file ONTT.2001-02-03T...." and a bunch of "delivered file ONTT.2001-02-03T...". OK?
- yes: It seems dhs is running. Are you sure the files are not in /data/raw/2001-02-03??? If no, you really are in trouble. You can try to restart dhs, but be sure you checked everything above before doing it.
- no: restart dhs, cf Normal Operation. Note that restarting dhs without restarting the Archive processes properly will cause troubles with the Archive: you must warn lsArchive. DO A BACKUP OF THE INSTRUMENT WORKSTATION AT THE END OF THE NIGHT.
- No: you should restart dhs. cf Normal Operation. Note that restarting dhs without restarting the Archive processes properly will cause troubles with the Archive: you must warn lsArchive. DO A BACKUP OF THE INSTRUMENT WORKSTATION AT THE END OF THE NIGHT.
- is dhs running properly now?
- yes: have a good night
- no: you are in trouble. You will have to transfer the data via FTP from wemmi/wsofi, and report that dhs is broken.
- 3- Raw data on Off-line workstation: wg5off
- 3.1 Are the raw data transferred on wg5off:
- cd /data/raw/2001-02-03
- ls : is the last file what you expect?
- yes: why are you reading this procedure?!?? your only excuse is if you are dealing with the reduced data: goto 4. Pipeline
- No: Do you have enough disk space?
- No: clean the disk and retry!!!
- Yes: continue this section
- 3.2 is dhsSubscribe-RAW running?
- check the dhsSubscribe-Raw light on nttexecDfsCheckAll
- Green? Strange. Consider it is Red, and continue
- Red? restart the dhsSubscribe-Raw with dataSubsciber, and retry. Does it work?
- yes: have a good night
- no: continue
- 3.3 is dhsSubscribe-RAW working properly?
- cd /data/msg
- more (or tail) DhsSubscribe-wg5off-RAW::wg5off::wg5dhs.2001-02-03 : you should get "...received file ONTT.2001-02-03...". OK?
- yes? Strange. Restart the dhsSubscribe-Raw with dataSubscriber anyway (cf Operation)
- No? any interresting error message? If yes, call lsArchive... If no, or if lsArchive/SWC not available, restart the dhsSubscribe-Raw with dataSubscriber anyway (cf Operation)
- is it working now?
- yes: have a good night
- no: kill and restart dhsSubsribe-Raw using the line commands (cf Operation)
- is it working now?
- yes: have a good night
- no: you are in trouble. FTP the data from wg5dhs.
- 4- Pipeline: wg5pl
- 4.1- Are the raw data transferred properly?
- cd /data/raw/200x-xx-xx
- ls : is the last image the one you expect?
- yes: OK, go to 4.2
- No: Do you have enough disk space?
- No: clean the disk and retry!!!
- Yes: continue this section
- 4.1.2 Is the dhsSubscribe running:
- Check the dhsSubscribe light on nttexecDfsCheckAll
- Green:
- quick recovery: goto 4.2
- Deep recovery: continue with 4.1.3
- Red: restart the dhsSubscribe on wg5pl
- 4.1.3 is the dhsSubscribe working?
- cd /data/msg
- tail DhsSubscribe.2001-02-03: you should get "received file ONTT...."
- yes: it is working. continue on 4.2
- no: restart the dhsSubscribe on wg5pl
- is it working now?
- yes: continue
- no: you are in trouble. Report the problem to lsArchive, and tell the astronomer to reduce his data by hand.
- 4.2 is the pipeline running?
- 4.2.1.Data Organizer: do you have a "DO" window that looks alive?
- yes:
- quick recovery: goto 4.2.2
- No or deep recovery: kill the DO window if it exist, and restart DO (cf operation)
- Is it working now?
- yes: continue
- no: you are in trouble. Report the problem to lsArchive, and tell the astronomer to reduce his data by hand.
- 4.2.2: Reduction Block Scheduler: do you have
- a RBS window that looks alive AND
- a Midas window that looks alive?
- yes:
- quick recovery: goto 4.3
- No, or deep recovery: kill the RBS and MIDAS windows if needed, and restart the RBS (cf operation)
- Is it working now?
- yes: continue
- no: you are in trouble. Report the problem to lsArchive, and tell the astronomer to reduce his data by hand.
- 4.3 Is the vcsolac running?
- check the vcsolac light for wg5pl on nttexecCheckAll.
- Green:
- quick recovery: goto 4.4
- Deep recovery: continue
- Red: restart vcsolac on wg5pl (Operation)
- is vcsolac running:
- psg vcsolac: you should get one process
- yes: continue
- no: restart vcsolac on wg5pl (Operation)
- is vcsolac working:
- cd /data/msg
- more (tail) VCSOLAC-wg5pl-PRO.2001-02-03: for each file, you should get "processing file...." and "delivered file...". OK?
- yes: vcsolac is really working, goto 4.4
- no: restart vcsolac on wg5pl (Operation)
- Is it working now?
- yes: continue
- no: you are in trouble. Report the problem to lsArchive, and tell the astronomer to reduce his data by hand.
- 4.4 is the dhs running:
- check the dhs light for wg5pl on nttexecCheckAll.
- Green:
- quick recovery: goto 4.5
- Deep recovery: continue
- Red: restart dhs on wg5pl (Operation)
- is dhs running:
- psg dhs: you should get a process that says "dhs -dhsdata... "
- yes: continue
- no: restart dhs on wg5pl (Operation)
- is dhs working:
- cd /data/msg
- more (tail) DHS-wg5pl.2001-02-03: for each file, you should get "received file....". OK?
- yes: dhs is really working. goto 5.
- no: restart dhs on wg5pl (Operation)
- Is it working now?
- yes: continue
- no: you are in trouble. Report the problem to lsArchive, and tell the astronomer to reduce his data by hand.
- 5- Reduced data on Off-line workstation: wg5off
- 5.1 Are the reduced data transferred on wg5off:
- cd /data/reduced/2001-02-03
- ls : is the last file what you expect?
- yes: why are you reading this procedure?!?? you should be observing or doing some maintenance...
- No: Do you have enough disk space?
- No: clean the disk and retry!!!
- Yes: continue this section
- 5.2 is dhsSubscribe-Reduced running?
- check the dhsSubscribe-Red light on nttexecDfsCheckAll
- Green? Strange. Consider it is Red, and continue
- Red? restart the dhsSubscribe-Red with dataSubsciber, and retry. Does it work?
- yes: have a good night
- no: continue
- 5.3 is dhsSubscribe-Reduced working properly?
- cd /data/msg
- more (or tail) DhsSubscribe-wg5off-RED::wg5off::wg5pl.2001-02-03 : you should get "...received file ONTT.2001-02-03...". OK?
- yes? Strange. Restart the dhsSubscribe-Red with dataSubscriber anyway (cf Operation)
- No? any interresting error message? If yes, call lsArchive... If no, or if lsArchive/SWC not available, restart the dhsSubscribe-Red with dataSubscriber anyway (cf Operation)
- is it working now?
- yes: have a good night
- no: kill and restart dhsSubsribe-Red using the line commands (cf Operation)
- is it working now?
- yes: have a good night
- no: you are in trouble. FTP the data from wg5pl.
3. DFS processes and control command
If you read the procedure till here and the dfs is still not working, it might be wise to call for help (Tatiana, Anton/Saul,Olivier, Michele Peron, in that order )3.1 vcsolac
To stop/clean vcsolac on the instrument workstations (wemmi/wsofi) or on wg5pl:
- on the workstation, type
cleanup-vcsolac
To start vcsolac:
- on the workstation, type
start-vcsolac
- check that vcsolac is running
psg vcsolac
3.2 Special cases
3.2.1 Pipeline
On wg5pl
- Data Organizer: start it from the wmmenu, or startDO
- RBS: start it from the wmmenu, or niceRBS
3.2.2 dataSubscriber
On wg5off, one should control and monitor the dhsSubscribe-RAW and dhsSubscribe-Reduced processes with the dataSubscriber.
3.2.2.1 To start dataSubscriber and the processes:
- Start it from wmmenu or type dataSubcriber
- enter Observer: service, Program Number: service, or the actual values for the current observer (but in this case, only the data from this observer will be transferred)
- [configure]
- [ok]
- [start raw] and [start reduced]
3.2.2.2 To stop the processes:
- [stop raw] and [stop reduced]
3.2.2.3 If starting and stopping the processes with dataSubscriber does now work:
- exit dataSubscriber
- [stop] the processes when you are asked while it the datasubscriber is exiting.
- stop the 2 processes by hand (see 3.3)
- restart the data subscriber: 3.2.2.1
- Is it working now?
- yes: OK, you are done
- no:
- kill the dataSubscriber and its processes (as described in this paragraph)
- instead of re-starting dataSubscriber, restart the processes by hand as described in next section.
3.3 DataFlow processes
The following table lists the information for each of all the DFS processes for the NTT. Note that processes in italic are restricted to lsArchive/SWC.
To operate a process:
- log on machine HOST as USER, for instance:
rlogin wg5off -l astro
- to stop the process: type CTRL_SCRIPT stop KEYWORD
- to start the process, type CTRL_SCRIPT start KEYWORD, for instance
dhsSubscribeControl start userRaw
- To check the logs of a process, cd LOG, and search for a file with the name of the process and the date of today, for instance
cd /data/msg
cat DhsSubscribe-wg5off-RAW::wg5off::wg5dhs.2001-02-03
USER HOST CTRL. SCRIPT KEYWORD LOG DATA NOTES archeso wg5dhs dhsControl olas /data/msg /data/raw pipeline wg5pl dhsSubscribeControl pipeline /data/msg /data/raw pipeline process pipeline wg5pl vcsolacControl pipeline /data/lists/reduced_olas /data/reduced pipeline process pipeline wg5pl dhsControl qc /data/msg /data/reduced pipeline process astro wg5off dhsSubscribeControl userRaw /data/msg /data/raw Started via dataSubscriber astro wg5off dhsSubscribeControl userReduced /data/msg /data/reduced Started via dataSubscriber
--oOo--
| Send
comments to: NTT Webmasters <nttweb@www.ls.eso.org> Last Update: MMM DD, YYYY (WHO) |