3.1 Calibrating data

In this tutorial chapter you will process data for the first time. We take the example of reducing raw science frames. This involves trimming, de-biasing, flat-fielding, illumination-correction, as well as the detection of saturated pixels, cosmic-rays and satellite tracks. Together with cold- and hot pixels these affected pixels are assigned a weight of zero in the weight map, which is also created. The results will be stored on the Astro-WISE dataservers and database. We assume that for now you want to store those such that only you can see these results. You can decide later to make the results accessible to other people.

3.1.1 Database projects and privileges

1. Data in the database is organized in projects. To see in which project you are currently working:

   awe> print(context.get_current_project())
   

   For example the output could be:

      Project:     OMEGACAM@VST  (id = 36)
      Description: All public data for OmegaCAM.  OmegaCAM is a 1 square
   degree wide field, optical, 16k X 16k pixel camera for the VLT Survey
   Telescope (VST) on Paranal Observatory.
      Instrument:  OMEGACAM
      Maximum Privileges: 4 (World)
      Current Privileges: 1 (MyDB)
   

   If you are doing this tutorial as part of a course you have been asked to produce the data in a certain project: TUTORIAL2015. You can change to this project by typing:

   awe> context.set_project('TUTORIAL2015')
   

   If you have not been asked to enter a certain project you can use the project you are in now. From now on, everything you create will be stored inside the chosen project. There will be other members of this project. You can set privileges to restrict who can view and access data within a project. A privilege of 1 means only you can view and access the data:

   awe> context.set_privileges(1)
   

   From here on, everything you create will be stored at privilege level 1 in the chosen project: only you can view/access the data. To allow other people in the project to see products, set privileges to 2 before you create them.

3.1.2 Processing science frames

1. Selecting and exploring RawScienceFrames Enter the following query that selects the raw science data objects, i.e., instantiations of the class RawScienceFrame, from the database.

   awe> query = \
        (RawScienceFrame.instrument.name == 'OMEGACAM') & \
        (RawScienceFrame.template.start == datetime.datetime(2012, 6, 1, 8, 51, 44)) & \
        (RawScienceFrame.is_valid > 0)
   

   To see how many raw science frames are found type:

   awe> len(query)
   160
   

   These are 5 OmegaCAM exposures, each consisting of 32 CCDs (32 RawScienceFrames are created for a single exposure).

   We select only the data of ccd#65 for now:

   awe> query = query & (RawScienceFrame.chip.name == 'ESO_CCD_#65')
   awe> len(query)
   5
   

   To obtain general meta-data of the last RawScienceFrame type:

   awe> raw = query.max('DATE_OBS')
   awe> raw.info()
   

   or only the observation dates, filter names, and maximum pixel value in the frame for all of them:

   awe> for r in query: print(r.DATE_OBS, r.filter.name, r.imstat.max)
   

   Download the FITS file:

   awe> raw.retrieve()
   

   Now that you have the file locally you can display it using your favorite image viewer.

   To visually inspect a frame with the help of a few analysis tools type:

   awe> raw.inspect()
   

   This brings up a graphical window. Hover your mouse over an object in the window and hit 'w': this shows a 3D wire-frame plot of the pixel values around the mouse pointer. Hit 'q' when you hover over it to close the graphical window. For a listing of all hot-keys type:

   awe> help(raw.inspect)
   

2. Reduce the RawScienceFrames that you found above.

   See http://www.astro-wise.org/portal/howtos/man_howto_calibrationpipelines/man_howto_calibrationpipelines.shtmlthis HOW-TO (http://www.astro-wise.org/portal/howtos/man_howto_calibrationpipelines/man_howto_calibrationpipelines.shtml) for a overview of how to process data in Astro-WISE. You will use standard recipes to reduce the RawScienceFrames. It is straightforward to adapt these or to write your own recipes, but this is beyond the scope of this tutorial.

   In the Astro-WISE environment you may choose the compute cluster (the Distributed Processing Unit or DPU) where the data reduction processes run. In that case no results are stored on your local machine. Created data (FITS files) are stored on the dataserver while meta-data is committed to the database.

   To de-bias and flatfield the above raw science data using the DPU you can use the ``Reduce'' recipe:

   awe> filenames = [raw.filename for raw in query]
   awe> dpu.run('Reduce', instrument='OMEGACAM', raw_filenames=filenames,
                commit=True)
   

   The commit=True switch ensures that your data is committed. Choosing commit=False will perform a ``dry run'': everything will be done except committing your results at the end. Alternatively, you can do the query and reduction at once via:

   awe> dpu.run('Reduce', instrument='OMEGACAM', template='2012-06-01T08:51:44',
                chip='ESO_CCD_#65', commit=True)
   

   You can view the status of your DPU job via

   awe> dpu.get_status()
   

   or by browsing the DPU server's webpage. See the links found on the: http://www.astro-wise.org/portal/aw_prompt.shtmlProcessing Grid page page (http://www.astro-wise.org/portal/aw_prompt.shtml).

   You can see how the processing went by retrieving the processing logs from the DPU.

   awe> dpu.get_logs()
   

   This command will not return anything until the process has been completed. The returned lines are also written to a single ("date+time".log) file in your local directory per awe-prompt session.

   To cancel DPU processing jobs:

   awe> for jobid in dpu.get_jobids(): dpu.cancel_job(jobid)
   

   Although it has advantages to do the processing of RawScienceFrames in parallel on a DPU it can sometimes be convenient to use your local machine. In this case the results are left in your current working directory. To de-bias and flatfield your raw science data found above using using local CPU you can use the same Reduce recipe as follows:

   awe> task = ReduceTask(instrument='OMEGACAM', raw_filenames=filenames,
                          commit=True)
   awe> task.execute()
   or...
   awe> task = ReduceTask(instrument='OMEGACAM', template='2012-06-01T08:51:44',
                          chip='ESO_CCD_#65', commit=True)
   awe> task.execute()
   

3.1.3 Inspect the results: ReducedScienceFrame

1. Locating the results. The results of above the process are de-biased, flatfielded science frames, which is a class of objects called ReducedScienceFrame in Astro-WISE. To select one of the ones you just created type:

   awe> qred = (ReducedScienceFrame.raw==query[0])
   awe> qred.project_only()
   awe> qred.user_only()
   awe> red = qred.max('creation_date')
   

   The first command queries for all ReducedScienceFrames which were created from the RawScienceFrame object query[0] in all projects you have access to. The second command narrows the selected ReducedScienceFrames to only those created within the project you are currently in. The third command zooms in on the subset of those that were created by you. The last command selects the ReducedScienceFrame that you created most recently. To do all this in a single command for all RawScienceFrames in query:

   awe> qred=[(ReducedScienceFrame.raw==raw).project_only().user_only().max('creation_date') for raw in query]
   

2. Inspecting the results. This can be done in similar fashion as you did for the RawScienceFrames above. For example:

   awe> qred[0].inspect()
   

3. Determine which calibration frames were used. As you did not create the bias frames and flatfields, the Astro-WISE environment selected those for you. To get general information on which ones were used:

   awe> qred[0].bias.info()
   awe> qred[0].flat.info()
   

   and to inspect one of the RawBiasFrames that were used to create the bias frame:

   awe> red=qred[0] #To allow Tab completion in next command....
   awe> red.bias.raw_bias_frames[0].inspect()
   

4. We have not configured anything beyond specifying the input frames in the previous examples. Find out which other parameters can be configured.
   
   See the http://www.astro-wise.org/portal/howtos/man_howto_configure/man_howto_configure.shtml for more details.configuration HOW-TO Calling the help() function on the Task that you are going to run gives a description of the arguments to the Task. The arguments are mostly query parameters which determine which input frames (RawScienceFrames in this case) are going to be reduced. In addition an argument ``pars'' can be specified which contains the processing parameters.

   awe> help(ReduceTask)
   

   Instantiate the Pars class with as argument the task, the pipeline identifier used in the dpu call or the class:

   awe> p = Pars(ReduceTask, instrument='OMEGACAM')
   awe> p = Pars('Reduce', instrument='OMEGACAM')
   awe> p = Pars(ReducedScienceFrame, instrument='OMEGACAM')
   

   By setting the instrument argument, instrument specific default processing parameters are used. Now call the show() method:

   awe> p.show()
   

   A parameter can be set as follows (use Tab completion):

   awe> p.ReducedScienceFrame.process_params.FRINGE_THRESHOLD_HIGH = 6.0
   

   Call the get() method to obtain the dictionary that you can supply to the task:

   awe> p.get()
   {'ReducedScienceFrame.process_params.FRINGE_THRESHOLD_HIGH': 6.0}