A Pipeline for basic processing of
ADONIS NIR data from SharpII+/Comic cameras

F. Marchis 2, D. Le Mignant 1

1/ ESO, La Silla-Santiago, Chile 
2/ presently at W. M.Keck Observatory, Hawaii


 

This page describes a pipeline for basic data reduction of the ADONIS data.

  • Introduction

The purpose of this web page is to present the data reduction routines written in C-Shell scripts by D. Le Mignant. These routines are based upon eclipse commands. They has been extensively tested and used and tested by the ADONIS team and visiting astronomers for more than 2 years. We wish now to make a general distribution of these commands for future and actual ADONIS observers. So, our goal here, is to describe the installation and give a how-to of this package.

For a complete description of the ADONIS Adaptive Optics system, its two camera SharpII+ and Comic and their associated prefocal optics optics, please refer to the official web page available here .

The present pipeline applies sky subtraction, flat-fielding and cosmetic corrections on your data and estimates their quality. It hasbeen developed for both ADONIS cameras and their prefocal optics.

This web page first describes the structure of the ADONIS data, the basic C-Shell routines to process them and some others to estimate the image quality. As a rule, any command line to be entered in the directory of the ADONIS cubes by the observer will appear in this document with that italic font.

First step of the data redction will require to get eclipse (Version 3.4) software installed on your computer. You will need to download the files: eclipse3_4.tar.gz and ScriptAdonis.tar. Unpack the tar files and follow the instructions in the README and INSTALL files. Please, refer to eclipse pages for more information. This web page will describe first the structure of the ADONIS data, the basic functions to process them and some extended functions to estimate their quality.

  • Structure of the ADONIS data

Both cameras are controlled by the ADOCAM software which save the frames taken into a FITS cube. The DAT tape you got after your observations must be read by the wdat eclipse command. You need to install and compile either only the wdat.c program, either the eclipse common package, plus its ADONIS extension both available here. By the way, the functions describe hereafter will need eclipse to work, so we advice you to install directly the eclipse software on your computer.

The dimension of the COMIC camera cube delivered by ADOCAM is 129*128*M (M the number of frames) and for SHARPII+ camera is 257*256*M. Therefore the ADONIS cubes do not comply 100% with the FITS standard. In fact, part of the time information is present in the images. To reformat the ADONIS FITS to a standard FITS, you can use the eclipse refits command.

For example,

> refits *.FITS will transform all the ADONIS cube to a standard FITS. Initial FITS cube are erased and replaced by the new one.



A FITS file is composed in two part:

  • An header which contains description of your data cube (dimension and number of plan), your observation (date and time of acquisition, field of view, filter, individual integration time...) and the observers.

    Important remark: there is no link between the TCS (Telescope Control System) and ADOCAM, so be sure that you wrote the coordinates of the telescope if your field of view is uncertain.

  • The data for every frame.

Check here to display a typical ADONIS FITS header.



How can you read the FITS header?

There is a dedicated function in eclipse called dfits; to use it just enter the following command:

> dfits *.FITS

If you want to read some specific keywords of the ADONIS header, use fitsort command

> dfits *.FITS | fitsort OBJECT NAXIS3 OP_FIEL OP_FILT DE_TIME OP_LAMB > InfoFits.txt

The display has been sent in file called InfoFits.txt.





  • Installation of the ADONIS Pipeline

The ADONIS Pipeline requires a series of scripts which are contained in the file ScriptAdonis.tar



It is composed of the following procedures:

adogain

adored

statis

extime

xtract

subclean

fotofast

badplan

based on eclipse Version 3.X commands.

If you are using an old version of eclipse 2.XX the same batches are still available under **_v2

A new version of the procedures for the eclipse 3.6 or higher is not yet available.... be patient.

Since these scripts are just shell commands,

just save them in your bin/ directory (select them pressing shift and click on the left button of your mouse)

type : chmod +x file

type : rehash

the help option is -h (ex: adored -h)



For RedHat Linux user, make sure that your.tcshrc file contains the following line

set path=($HOME/bin)



To check if the installation is correct, open a new window and type:

> which adored

It must appear:

>/home/yourname/bin/adored







  • Flat-Field and Badpixel map creation

To create a response map of the SHARPII+ camera (taking into account all the optics, from the camera to the telescope) among various filters and field of view, the easiest way is to take some high signal to noise images (flux on each frame > 4000 ADU) of the daylight sky and some DARK frame with the same integration time.

A good approximation of the COMIC response map can be obtained performing the same operation but observing the integration sphere flat-field mounted on the bench.



The Flat-Field map can be derived easily using the adogain command. To get a help enter:

> adogain -h

Usage: adogain [-v] <input cube> <dark>

-------------------------------------------------------------------

--- adogain is a script dedicated to adonis + sharp data

--- - Gain map an badpixel map name are generated automatically

--- from the input cube name, band and pixel scale

--- - The gain map is computing from a cube in a band

--- and the corresponding dark cube.

--- - The badpixel map is derived assuming that for sharp

--- the badpixel number is greater than 350 - see script -

--- - Badpixel in the output gain map are tagged to 1.00

--- Parameters:

--- -----------

--- Input cube: sky flat cube in a given band

--- dark: dark cube with the same integration time



This procedure calculates the average on input cube and dark cube, subtract the dark frame to the input frame and normalize the resulting frame. bad pixel map is also build by detection of the extreme values in the normalize frame.

It has been found to be the easiest and fastest way to create the flat-field frame with both cameras. User can also make them at twilight like visible observation and processes them by the flat procedure of eclipse, but we did not observe any significant improvement in comparison with the adogain process, ONLY in the case that the S/N is high enough on the sky frames.



The data are saved on the disk using the following nomenclature: gainYYMMDDwavescale.fits for the flat-field frame and badYYMMDDwavescale.fits for the bad pixel map.

Example: gain991025SK100.fits is a flat field frame taken on the 25th of November 1999 in SK band with the 100 mas pixel scale.



Fig 1:An example of the Flat-Field and bad pixmap of SHARPII+ (left) and COMIC (right) camera



Click on the image to get a better display


Click on the image to get a better display


SHARPII+ Flat Field in SK 100mas
Flat Field in SK band (100mas) with Sharp

COMIC flat field in L' 100mas
Flat Field in L' band (100mas) with Comic




Sharp bad pixel Map


COMIC bad pixel Map


  • Cosmetic data processing

The cosmetic data processing of the ADONIS images is the classical data reduction applied on Infrared cameras. Mainly subtraction of the sky, division by the flat-field and bad pixel removal. We present two automatic procedures which process the data thanks to information written in the FITS header.



Subclean procedure:

The First one called subclean if specifically built for observations which have been performed in "stearing" mode. In this case, the sky is taken starting a new acquisition after moving the telescope. Coronagraphic or short band observations are generally performed by this way. Subclean is easy to use:

>subclean -h

Usage: subclean [-v] <Cube_Star> <Cube_sky> <gain_map> <bapixel_map>

-------------------------------------------------------------------

---- Subclean uses Eclipse V3 commands: ccube, deadpix...

---- Subclean compute a median from the sky cube

---- Then subtract the median sky to the star planes

---- Correct for flatfield and badpixel

---- The ouput files are 1: a cleaned star cube

---- 2: the average image of the cleaned star planes

---- 3: the median image of the cleaned star planes

-------------------------------------------------------------------

Example: subclean tco991025_0023.FITS tco991025_0025.FITS gain991025SK100.fits bad991025J100.fits

Three files are generated after processing:

- *red.fits which is the cleaned star cube

- *redavg.fits an average frame of this cleaned star cube

-*redmed.fits a median image of this cleaned star cube

The previous example will create three files called tco991025_0023_red.fits, tco991025_0023_redavg.fits and tco991025_0023_redmed.fits

Fig 2: A brief summary of the data processing performed by subclean on COMIC images.









Adored procedure:

In the case that you used the On/Off mirror to perform your observations, a cube of data containing both On and Off frames is saved in one *.FITS file. This chopping observations can be processed using the adored function which also works in the case you used the internal chopping observing mode. To use adored you must indicate 3 files and 1 parameter:

> adored -h

Usage: adored [-v] <Cube> <flag> <gain_map> <bad_map>

--------------------------------------------------------

---- Chopping mode is relative, flag=0

---- Chopping mode is reality, WITH micro-scanning flag=1

---- Step, cycle and chopping pattern are taken from fits header

---- This script is based on Eclipse version3 commands.

--------------------------------------------------------

Example: adored tco_991025_0434.FITS 1 gain991025J35.fits bad991025J100.fits

will generate 2 file cubes called tco_991025_0434_red01.fits and tco_991025_0434_red10.fits


Fig 3: Short description of the adored procedure applied on a SHARPII+ image.

IMPORTANT: A bug due to an eclipse command has been reported on eclipse V3.1 and latter (until 3.4).

On a big cube (>50 frames) the eclipse command extract does not work properly. We wrote a temporal new version of adored named adoredcor.

You also need to install on the directory from where you run the UNIX command , a small awk program called "prothese".

The use of adoredcor is similar to adored. Contact F. Marchis (fmarchis@eso.org) if you need help.





  • Other procedures for basic reduction

xtract and xtract64 procedures:

These two procedures are used to extract from a reduced cube a smaller cube of 128*128 (for xtract) and 64*64 pixels (for xtract64) around a star which is detected automatically. This function is useful to discard the negative part of each frame after reduction of a internal chopping cube by adored. One parameter defining the sigma detection level is needed.

> xtract -h

Usage: xtract <Cube> <sigma_detection>

----------------------------------------------------

---- Extract 128x128 images centered on the star..

----.. detected in the first plane

---- Enter sigma_detection for the star

----------------------------------------------------
 

Example: xtract tco_991025_0324_red01.fits 100

will generate a cube file tco_991025_0324_red01_128.fits of 128*128*M


Badplan procedure:

>badplan -v

Usage: [-v] <input cube>

-------------------------------------------------------------------

--- This script is useful to remove the corrupted plans in a data cube

--- Good plans and corrupted plans are concatenated in different cube

--- Then the information is recorded in a separate file (data_filename.txt)

-------------------------------------------------------------------

This command is useful to separate bad frames in a huge cube of data. It can be only apply on raw data (*.FITS). Selection of good frames is done using the stcube eclipse function.

The command statis is very useful to calculate easily the Strehl ratio and FWHM on a PSF cube after complete basic data reduction.

>statis -h

Usage: statis [-v] <Cube> <Band> <PixelScale> <threshold> <sigma_detection>



Example: statis KGB_991203_0227_red01_128.fits SK 0.035 3 40

will generate a file called KGB_991203_0227_red01_128.stat which contain the Strehl Ratio and FWHM for the PSF on every frame of the cube.



  • Conclusion

These UNIX batch commands are very powerful and easy to use and we encourage future ADONIS users to process their data using them.. A complete processing with the adored function of a SHARPII+ data cube of 400 frames will take less than 5 minutes on a LINUX PC machine and less than 20 min on a SUN Ultra Sparc 1 station. They have been written taken advantage of our knowledge in ADONIS data processing and thus are optimized for this kind of data. Feel free to report us your comments, ideas and improvements.

A new version of the script will be soon available for the eclipse 3.6 and newer version.



© 2000, F. Marchis- ESO

http://sc6.sc.eso.org/~fmarchis/calib/basicprocess
Last update: June 7th, 2000

Contact :

F. Marchis, marchis@eso.org, ESO, La Silla, CHile

D. Lemignant, dlemigna@eso.org, ESO, La Silla, Chile