1.1 HOW-TO Introduction

1.1.1 History

The Astro-WISE Environment (AWE) was created by The Astro-WISE Consortium, a partnership of:

and coordinated by OmegaCEN-NOVA. AWE was conceived as the solution to handle the vast amounts of astronomical data generated by all-sky surveys, particularly those to be observed with OmegaCAM on the VLT Survey Telescope (VST) on Cerro Paranal in Chile.

While waiting for OmegaCAM and VST completion, AWE has been expanded to include data from many different sources. Currently supported data sources include the Wide-Field Imager (WFI) on the 2.2m MPG at La Silla, the Wide-Field Camera (WFC) on the INT at La Palma, and SuprimeCAM (SUP) on Subaru at Mauna Kea. The public portions of these datasets can be browsed at the Supported Data Sources section of the Astro-WISE Portal.

1.1.2 Basic Philosophy

  • General
    • The Astro-WISE Environment (AWE) is an information system for the scientific analysis of extremely large datasets. It utilizes federated databases and dataservers, and parallel compute clusters to manage these vast amounts of data.
    • It was originally designed and developed specifically for astronomical wide-field imaging surveys, but has been used for the analysis of large datasets of handwritten archives and can be applied to any application involving very large datasets.
    • AWE is a federated system: data can be made any one-location in the federated system, but used everywhere in the federated system. This allows collaboration between diverse working groups.
      NOTE: AWE is currently operational in The Netherlands (Groningen and Leiden), Germany (Bonn and Munich), and Itlay (Naples)

    • Raw data is sacred in AWE. All data in the system is derived from raw data and can be traced back to the raw data within the system. This facilitates on-the-fly re-processing (OTFR) whenever improved methods or data is available.

    • All data in AWE is linked via backward chaining. Thus system is an all-in-one system: ALL input and output of processes are stored in the information system.

    • In AWE, the most recent product of a series of processes is considered the best: newer is better. Processes in AWE automatically use the latest versions of calibration files and software codes, which are both assumed to be the best.

    • The processing of data in AWE is split up such a way that it is embarassingly parallel. For astronomical data, this means all processes are per CCD.

    • AWE makes extensive use of the Python programming/scripting language in all its aspects.

  • Object Model
    • Data are represented as Python objects with attributes corresponding to both pixel-data and meta-data, and methods corresponding to actions to perform on the object and its attributes.
    • Each object is considered a target that can be ``made'' with Tasks (e.g., found in the astro/recipes directory ) which set mandatory dependencies and run the target's make() method. Every make() fills in the newly instantiated (i.e., created) object. For example:

      awe> bias = BiasFrame()

      instantiates an EMPTY bias frame object.

      awe> bias.make()

      completes/fills in the object's attributes such as the observation date, pixel and header data, data statistics, etc.

    • With the Target Processor, the concept of a UNIX ``make'' is duplicated where all dependencies are checked for up-to-dateness (this includes existence) and will have their own make() method run if they are not up-to-date. This occurs recursively back to the raw data.

      NOTE: This recursive ``making'' of objects does not extend to the Tasks as the dependencies are only checked for existence.

    • Each object has verify(), inspect(), and compare() methods. These ensure optimum data quality.
  • Code Access
    • Users have direct access to the Python code base via CVS checkout at http://cvs.astro-wise.org/
    • Users have the ability to add their own code or modify certain existing code.
    • The core parts of the system should not be modified by users.
    • Recipes can be modified for convenience in the users own checkout.
    • All the other parts of the system should only be modified if there is a bug, and preferably by the maintaniner of that part.
    • See §[*] for more information.

  • Documentation
  • Services
    • See §[*] for the various web-based interfaces to the system.

1.1.3 Hardware

In the architecture of the Astro-WISE system, three main components are identified: a file-server, a database, and a high-performance compute cluster.

The file-server stores FITS-files, while the database keeps track of the relations between these files and their processing history. It is also through this database that decisions are made about which files to retrieve during the various processing steps. The compute cluster is used to process the data. During processing, requests are made to the database for the raw science data and for the necessary calibration files, which are then retrieved from the file-server.

NOTE: The only files the user has direct access to locally (e.g., current directory) are data products retrieved and/or created during processing on the local machine. Normal processing using the compute cluster will leave no such data products, but only log files at most.

1.1.4 Software

The software consists of code written in Python, and includes an interactive command line environment (see awe-prompt HOW-TO), along with web services for viewing and editing the database (see Web Services). On a lower level a number of existing C programs are used, such as SExtractor, LDAC, SWarp, and Eclipse.


1.1.4.1 Python and Object Oriented Programming

The code for the Astro-WISE system is written in Python, which is a language highly suitable for Object Oriented Programming (OOP). Within the OOP style in which the code is written, classes are associated with the various conventional calibration images, data images, and other derived data products. For example, in our system, bias exposures become instances of the RawBiasFrame class, and twilight flats become instances of the RawTwilightFlatFrame class. These instances of classes are the ``objects'' of OOP.

Classes may have incorporated methods and attributes. Methods perform a task on the object they belong to, while attributes are properties such as constants, flags, or links to other objects that may be needed by methods. In Astro-WISE various recipes have been coded that control the creation of instances of these classes. There may be different ways to create an instance of a class depending on which attributes are set to what values, and which methods are used. A ColdPixelMap object, for example, can be instantiated from the database (i.e. as the result of a query or search) or it can be created by using its ``make()'' method. In the latter case the ColdPixelMap can be derived either from a DomeFlatFrame or a TwilightFlatFrame, depending on which of those two objects are specified as the ``flat'' attribute of the ColdPixelMap object.

Within the Object Oriented Programming style, inheritance is an important and powerful concept. Classes can inherit attributes and methods from other classes. For example, both the DomeFlatFrame and TwilightFlatFrame classes are derived from the base class BaseFlatFrame; both are flat-fields afterall. Any method or attribute defined by the BaseFlatFrame class is inherited by both classes and both classes are free to redefine (this is called polymorphism) those methods or attributes and even add new ones as needed.

The bottom line is that Astro-WISE turns data into objects that are instances of Python classes with attributes and methods that can be inherited.

There is a significant amount of on-line documentation available for Python. Please see the Python web site http://www.python.org for further background on Python and for Python tutorials.


1.2 Web Services

1.2.1 Database Viewer

A web-service is available to view the database content and pixel data. It can be found at the following address:

http://dbview.astro-wise.org/

Help pages are provided by the webservice itself.

1.2.2 Database ``Editor''

For a limited number of operations editing of database values is allowed. Specifically, it is possible to change valid ranges of calibration data (timestamps), and flags, to disqualify bad data. A special web-service tool to change these values can be found at the following web-site:

http://calts.astro-wise.org/

Help pages are provided by the webservice itself.

1.2.3 Processing Web Interface

The Target Processor is the culmination of all the benefits of the Astro-WISE system. It allows processing of a target (an end data product such as a ReducedScienceFrame or SourceList) and any of its dependencies that require it, on a parallel (e.g., compute cluster) or on a single (e.g., local machine) host. The dependency chain is followed back towards the raw data (backward chaining) to make sure only those objects requiring processing are actually processed.

http://process.astro-wise.org/

A web-based version of the awe-prompt command-line interface also exists for convenience (account required).

http://process.astro-wise.org/AWE

1.2.4 Image Handling Services

Services involving image sections (dependency cutouts, RGB generator, etc.) can be found in the IMGView service:

http://imageview.astro-wise.org/

1.3 Further Websites

The OmegaCAM web site:
http://www.astro.rug.nl/~omegacam/
The Astro-WISE web site:
http://www.astro-wise.org/
The Astro-WISE Web Services web site:
http://portal.astro-wise.org/



page generated Tue Apr 4 14:29:37 CEST 2017