1.1 General

1.1.1 Introductory Material

  • Q: What is Astro-WISE?

    A: Astro-WISE for Astronomical Wide-field Imaging System Europe, is an E.U. consortium and associated information system dedicated to the handling of wide-field image data resulting from large astronomical surveys. Please see the main page for a complete description.

  • Q: What is the Astro-WISE Environment?

    A: The Astro-WISE Environment, or simply AWE, is the information system built to handle this flood of astronomical data. Start here to learn more about it.

  • Q: What language is the Astro-WISE Environment written in?

    A: The Astro-WISE Environmentis written mainly in the Python programming/scripting language. There are also Python utility modules written in C (see the HOW-TO Introduction), but these are wrapped in Python to interface with the system.

  • Q: What kind of web-services does AWE offer?

    A: See the Introduction HOW-TO for a complete list of what AWE offers.

1.1.2 Getting Started

  • Q: I can't access anything but the basics in the AWE, nor can I access the database. What's wrong?

    A: Do you have an AWE account? If not, please contact an Astro-WISE representative.

  • Q: I try to access the anonymous CVS server for a checkout, but it appears to need a password. What is it?

    A: Anonymous CVS requires a special password and is not available to just anybody. Contact an Astro-WISE representative for help. Anonymous CVS is read-only access to the software. When ready to contribute to the code-base, you will need a full account. Again, see an Astro-WISE representative.

  • Q: I keep seeing references to $AWEPIPE. What is it?

    A: $AWEPIPE is the environmental variable that should point to your ``awe'' checkout, the code-base for AWE (e.g., 1#1 mathend000#/awe/).

  • Q: How do I start AWE?

    A: There are many options, but the typical one is to simply use AWE on a properly configured system:

    ...]$ awe
    Python 2.3.5 (#4, Aug 15 2005, 11:45:46)
    [GCC 3.4.3 20050227 (Red Hat 3.4.3-22.1)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
                   Welcome to the Astro-WISE Environment

    You can also temporarily modify environmental variables that AWE uses so that AWE starts with a different configuration:

    ...]$ env AWEPIPE=~/test-awe/ awe
    ...]$ env database_engine= awe

    The first example causes AWE to use a code-base located in a specified directory (1#1 mathend000#/test-awe/ here) and the second disables use of the database (if you need to work offline for some reason).

1.1.3 Documentation

  • Q: I'm a new user. Where should I look for documentation?

    A: The first place any new user should look for documentation is the HOW-TOs

  • Q: What are docstrings?

    A: Docstrings are the inline comments added to Python code that allow for an automated documentation system called PyDoc. Docstrings are contained within triple quotes:

    '''this is a docstring'''
    """this is also a docstring"""
    # this is just a comment

1.1.4 Concurrent Versions System (CVS)

  • Q: What is the address of the Astro-WISE CVS server?

    A: cvs.astro-wise.org

  • Q: What is the proper way to fully update a code-base?

    A: First, make sure you are in the top-level directory (or whatever directory you want fully updated) and issue the following command:

    ...]$ cvs -q update -dPA


    suppresses excessive verboseness (option to cvs)
    create any missing directories (option to update)
    prune empty directories (option to update)
    reset any ``sticky'' tags (option to update)

    If you chose a specific tagged version, that tag becomes sticky. The last option overrides this stickiness. See the cvs(1) man page for complete information on CVS.

  • Q: I am installing the Astro-WISE system on my local machine, but when I try to install < mathend000#insert program name here> mathend000#, I get the comment that some file or directory is missing.

    A: are you sure that you retrieved every single sub-directory of the Astro-WISE system during checkout? Using a simple cvs co can sometimes result in some sub-directories not being retrieved. If you suspect that something like this might be the problem, go to the directory where the specific sub-directory should have been located and type cvs update -d.

  • Q: How do I switch between the latest $AWEPIPE version and the AWBASE version?

    A: To switch to the most recent check out use

    ... awe]$ cvs -q update -dPA

    and to switch back to the AWBASE version, use

    ... awe]$ cvs -q update -r AWBASE -dP

    in the `awe' directory.

1.1.5 Data Preparation

  • Q: I am about to go observing with an AWE-supported instrument. Is there anything special that I should do?

    A: Please look at the Observations Scheduling HOW-TO for guidelines.

  • Q: I want to process the data I took on < mathend000#insert favorite instrument> mathend000# with AWE. What do I need to do?

    A: AWE was created specifically for OmegaCAM and will handle OmegaCAM data ``out-of-the-box''. There is also support for the instruments listed on the Supported Data Sources page. If your instrument is not there, contact an AWE representative for verification and have a look at the New Instrument HOW-TO to get started.

1.1.6 Ingesting

  • Q: What is data ingestion and how do I do it?

    A: In order for raw data to be processed in the Astro-WISE system, it must first be ingested, or imported into the system. Please see the Data Ingestion HOW-TO for a complete description.

  • Q: How can the data be categorized for ingestion in AWE?

    A: Data in AWE is catagorized by purpose:

    bias frames designated to determine the instrument read noise
    a raw bias or zero second exposure frames
    a frame taken to measure the dark current of the instrument
    a specific series of frames designed to determine the gain of the instrument
    a raw flat frame taken from a screen mounted within the dome enclosure
    a raw flat frame taken at twilight on the sky
    any frame taken for the purpose of science (excluding photometric calibration)
    a science frame taken with the specific purpose of photometric calibration

  • Q: After I ingested my data, all the filenames were different. What happened?

    A: The filenames were converted to AWE cannonical names of the form:


    The first from is for multi-extension FITS images, the second is for single-extension FITS images where n is the extension number.

1.1.7 Dates and Times

  • Q: How are observation dates defined in AWE?

    A: An observation night is based on the local date at sunset and not on UTC. See the Dates and Times HOW-TO for more information.

  • Q: How are dates stored in the database?

    A: In UTC only.

1.1.8 Parallel Processing

Q: What is the parallel processing interface in AWE?

A: The DPU (Distruibuted Processing Unit) is used to run tasks on the parallel compute cluster. It is initialized when AWE starts:

...]$ awe
Importing Astro-WISE packages. Please wait...

Initializing Distributed Processing Unit...

Current profile:

To initiallize it manually, use this method:

awe> from astro.recipes.mods.dpu import Processor
awe> my_dpu = Processor(Env['dpu_name'])

Q: How do I find the instruments, task identifiers, and options supported by the DPU's run method: dpu.run()?

A: This information is somewhat hidden. The supported instrument list can be seen by querying the HeaderTranslatorFactory:

awe> from astro.instrument.HeaderTranslatorFactory import supported_instrument_list
awe> print supported_instrument_list

Task identifiers can be found like this:

awe> from astro.recipes.mods import Pipeline
awe> for id in Pipeline.get_available_sequence_identifiers(): print id

And the options can be printed in this way:

awe> from astro.recipes.util.ArgumentParser import ArgumentParser
awe> for opt in ArgumentParser().opts: print opt

This gives you a list of tuples of the form (short_opt, long_opt, definition).

1.1.9 awe-prompt

  • Q: What is namespace and how can I see it?

    A: Simply speaking, namespace is all modules, classes, attributes, and methods available in the current scope (i.e., level). If the module, class, attribute, or method you need is not visible in the current namespace, then it cannot be used. It is either loaded within the namespace at a different scope, or it is not there at all. The builtins dir() and help() allow you to explore the namespace.

    dir() without any arguments gives the namespace of the current scope. It typically will give Python builtins and any modules pre-loaded at startup or during the current session. Calling dir() with a module, class, attribute, or method as the argument will give you the namespace at that level or an arbitrary level:

    awe> dir()
    ['__builtin__', '__builtins__', '__doc__', '__file__', '__name__', '__version_
    _', 'astro', 'atexit', 'os', 'pydoc', 'readline', 'rlcompleter', 'sys', 'users
    awe> dir(os)
    ttyname', 'umask', 'uname', 'unlink', 'unsetenv', 'utime', 'wait', 'waitpid', 
    'walk', 'write']
    awe> dir(os.write)
    ['__call__', '__class__', '__cmp__', '__delattr__', '__doc__', '__getattribute
    __', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__'
    , '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__str__']

    help() can be used in a similar way, but it gives all the docstrings recursively for the module, class, or method, or for the parent class of the attribute that was given as the argument.

  • Q: When importing something from the awe-prompt, I get strange error messages through my screen. I know that the software is okay. What is the deal?

    A: Make sure that you are not importing something from within the awe directory structure. Due to some strange Python things concerning packages, this can lead to completely intractable error messages.

  • Q: One of the Python recipes generates an error and I can not figure out what is wrong. What should I do?

    A: Sometimes problems arise because of old compiled Python files (*.pyc); there may be leftovers of Python files that no longer exist. Remove all these files and try again.

  • Q: What are some usefull packages and modules available at the awe-prompt?

    A: See the awe-prompt HOW-TO.

1.1.10 Queries

  • Q: What are queries and how do I use them?

    A: Queries in AWE are queries to the database from Python with the purpose of returning some needed information. The Queries HOW-TO containes a complete explaination of the use of queries in AWE.

  • Q: What's the deal with the select() method?

    A: The select() method is simply a convenience method to find the most recent, valid version of a given frame. Use the Python builtin help() on a particular select method to find out exactly how it makes this choice.

1.1.11 Process Parameters

  • Q: Which process parameters can be configured and where?

    A: Generally speaking, all process parameters can be configured from the awe-prompt prior to running a make or executing a task. The Configuration HOW-TO gives a complete description of the various methods for process parameter configuration.

1.1.12 Context

  • Q:


  • Q:


  • Q:


  • Q:


  • Q:


1.2 Astro-WISE Environment

  • Q: I want to run SExtractor on my images. How do I do that ?

    A: There are basically three ways to run SExtractor in the system. The most direct one would be to import `Sextractor' from `astro/external' and run `Sextractor.sex(image)'. You can also set the SExtractor configuration through this interface, and modify the list of output parameters. A second way to run SExtractor would be using the `sex' method of BaseFrame or children thereof. To be concrete, instantiate a ScienceFrame object and call the `sex' method. The third, and in our paradigm the most correct, way to run SExtractor, is to invoke the `make' method of a Catalog object. The frame you want to extract sources from is given as a dependency to the Catalog object. For the configuration of SExtractor and to modify its outputs, you need to provide the Catalog object with two additional dependencies: a SextractorConfig object, and a list of parameters. Note that this last way of creating a SExtractor catalog uses the first method. Also note, that the output of running SExtractor is in LDAC fits format.

  • Q:


  • Q:


  • Q:


1.3 AW Tutorials

  • Q:


  • Q:


  • Q:


1.4 Calibration

  • Q:


  • Q:


  • Q:


1.5 Image Pipeline

  • Q:


  • Q:


  • Q:


1.6 Visualization

  • Q:


  • Q:


  • Q:


1.7 Development

  • Q:


  • Q:


  • Q:


page generated Tue Apr 4 14:29:24 CEST 2017