/portal/howtos/man_howto_tutorial_introduction/man_howto_tutorial_introduction.shtmlprevious | /portal/howtos/man_howto_tutorial_introduction/man_howto_tutorial_introduction.shtmlhome | /portal/howtos/man_howto_tutorial_science/man_howto_tutorial_science.shtmlnext |
In order to start up the command-line interface, it is necessary to modify your shell environment so the AWE software can be found and the system can log you in to the database. The former is done using the module framework, the latter by creating a configuration file for AWE:
On the network of the Kapteyn Astronomical Institute:
If you're using ``csh'' or ``bash'' add the following line to the file
1#1
module load awe
It is necessary to obtain a database username and password in order to login to and write (``commit'') to the database. Please ask your http://www.astro-wise.org/dbas.shtmllocal Astro-WISE DBA for an account.
In your home directory make a directory .awe (starting with a dot) and in this subdirectory make a file called ``Environment.cfg'', readable only to yourself:
...]$ cd ...]$ mkdir .awe ...]$ cd .awe ...]$ touch Environment.cfg ...]$ chmod a-rwx,u+rw Environment.cfg
Then add the following lines to that file replacing your username (e.g.,
awjkennedy) and password (e.g., IwonIn1960) in the appropriate places. The
line project
ensures the awe-prompt always starts in context ALL
(what context is will be explained later in the tutorial). The line
privileges : 1
ensures that it starts within the private space of
that context ('MyDB').
[global] database_user : awjkennedy database_password : IwonIn1960 project : ALL privileges : 1
Start the awe-prompt from your shell:
...]$ awe
Check that your username and default project are used. In the above case that means the welcome message ends with a message like this:
Current profile: - username : awjkennedy - database : db.astro.rug.astro-wise.org - project : ALL - current privileges : 1 (MyDB) awe>
you now have access to the database. Quit awe by hitting Ctrl-d.
awe
at the command line. The
awe-prompt is the standard Python interpreter plus a few additions.
The Python interpreter has built-in functions and classes:
awe> dir(__builtins__)
If you are not familiar with the Python language and/or object oriented programming it is a worthwile to browse https://docs.python.org/3/tutorial/index.htmlthis Python tutorial. It will make the rest of the tutorial much easier to grasp. Many object oriented programming concepts (especially the chapter on classes) are crucial concepts at the awe-prompt. The built-in functions of Python contain basic functions such as ``range'', ``len'', ``int'', ``float'', ``dir'' and ``help'' that can be useful at the awe-prompt. To illustrate this, the following example calculates element 19 of the http://en.wikipedia.org/wiki/Fibonacci_numberFibonacci sequence using Python syntax:
awe> int(round( ( (1.0+5.0**0.5)**19 - (1.0-5.0**0.5)**19 ) / (2.0**19*5.0**0.5) )) 4181
awe> dir()
awe> help() awe> help() Welcome to Python 3.5's help utility! If this is your first time using Python, you should definitely check out the tutorial on the Internet at http://docs.python.org/3.5/tutorial/. . . help>Follow the instructions to find help on pretty much anything by typing the name of it. Use quit or Ctrl-d to exit. It is usually convenient to get help on a specific module, function, or even object. To see what dir() does exactly, for instance, type:
awe> help(dir) # the screen clears and shows the following: Help on built-in function dir in module __builtin__: dir(...) dir([object]) -> list of strings If called without an argument, return the names in the current scope. . .
awe> dir(Chip) ['__class__', '__del__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_delete', '_inverses', '_publish', 'as_dict', 'commit', 'copy_persistent_properties', 'dbproxy', 'get_creator', 'get_inverse_properties', 'get_persistent', 'get_project', 'info', 'inverse_objects', 'inverse_query', 'name', 'object_id', 'persists', 'pickle_id', 'pixelsize', 'recommit', 'select_all', 'update_header']
awe> dir(RawTwilightFlatFrame) ['DATE', 'DATE_OBS', 'EXPTIME', 'LST', 'MEAN_HIGH', 'MEAN_LOW', 'MJD_OBS', 'NAXIS1', 'NAXIS2', 'OBJECT', 'OBSERVER', 'OVSCX', 'OVSCXPRE', 'OVSCXPST', 'OVSCY', 'OVSCYPRE', 'OVSCYPST', 'PROCESS_TIME', 'PRSCX', 'PRSCXPRE', 'PRSCXPST', 'PRSCY', 'PRSCYPRE', 'PRSCYPST', 'STATUS_COMPARE', 'STATUS_INSPECT', 'STATUS_MAKE', 'STATUS_VERIFY', 'UTC', '_IS_ABSTRACT', '_IS_CAL', '_IS_CONFIG', '_IS_RAW', '_IS_SCIENCE', '_IS_SEQ', '_IS_SUPPORT', . .The namespace lists both properties (e.g., observation date DATE_OBS, exposure time EXPTIME) and methods (e.g., inspect, display) of the class.
['DATE', 'DATE_OBS', 'EXPTIME', 'LST', 'MJD_OBS', 'NAXIS1', 'NAXIS2', 'OBJECT', 'OBSERVER', 'OVSCX', 'OVSCXPRE', 'OVSCXPST', 'OVSCY', 'OVSCYPRE', 'OVSCYPST', 'PRSCX', 'PRSCXPRE', 'PRSCXPST', 'PRSCY', 'PRSCYPRE', 'PRSCYPST', 'UTC', 'chip', 'creation_date', 'extension', 'filename', 'filter', 'globalname', 'imstat', 'instrument', 'is_valid', 'object_id', 'observing_block', 'overscan_x_stat', 'overscan_y_stat', 'prescan_x_stat', 'prescan_y_stat', 'process_params', 'process_status', 'quality_flags', 'raw_fits_data', 'template']In this list ``chip'' points to another class: the class Chip we just described. To list its persistent properties type:
awe> RawTwilightFlatFrame.chip.get_persistent_properties() ['name', 'object_id'] or awe> Chip.get_persistent_properties() ['name', 'object_id']
awe> help(RawTwilightFlatFrame) Help on class RawTwilightFlatFrame in module astro.main.RawFrame: class RawTwilightFlatFrame(RawFrame) | A base class for persistent FITS file objects | | BaseFrame inherits from the persistent DataObject class and the | ProcessTarget mixin. DataObject provides data retrieval and | storage facilities for the FITS files. All BaseFrame objects are | ProcessTargets (have a make method). | | BaseFrame itself is intended to be an abstract baseclass. | | A BaseFrame instance defines a number of attributes, | | image -- An eclipse.image object (default None), read if needed, | or by load_image() | header -- An darma.header object (default None), read if needed, | or by load_header(), or created by build_header() | history -- A list of strings containing the history of the object. | currently not read from the header, but used by | build_header() | | Method resolution order: | RawTwilightFlatFrame | RawFrame | astro.main.BaseFrame.BaseFrame | common.database.DataObject.DataObject | common.database.DBMain.DBObject | astro.main.ProcessTarget.ProcessTarget | common.database.DBMeta.DBMixin | common.main.ProcessTarget.ProcessTarget | astro.main.OnTheFly.OnTheFly | common.main.OnTheFly.OnTheFly | builtins.object ... ... | ---------------------------------------------------------------------- | Data descriptors defined here: | | DATE | UTC date the original data file was saved [None] | | DATE_OBS | UTC date at the start of the observation [None] | | EXPTIME | Total observation time [sec] ..... Hit "q" to exit the help page.Here the difference between
DATE
and DATE_OBS
becomes
clear. The help tells you much more than just the properties. The first
line tells you that the code for this class is stored in module
astro.main.RawFrame (how to find the code will be explained later). It is
followed by a definition of RawTwilightFlatFrames. The explanation of the
other lines is beyond the scope of this tutorial.
NB: enter the following 2 lines on 1 line of the awe prompt: awe> query = (RawTwilightFlatFrame.instrument.name == 'OMEGACAM') & (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1)) NB: you can enter the following command on 2 input lines by continuing the line with '\': awe> query = (RawTwilightFlatFrame.instrument.name == 'OMEGACAM') & \ (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1)) or by using an extra set of parentheses: awe> query = ((RawTwilightFlatFrame.instrument.name == 'OMEGACAM') & (RawTwilightFlatFrame.DATE_OBS < datetime.datetime(2011,9,1))) awe> len(query) 6240
You can select one of these RawTwilightFrames using an index and see when it was observed exactly:
awe> raw = query[10] awe> raw.DATE_OBS datetime.datetime(2011, 8, 10, 22, 49, 22)or more directly:
awe> query[10].DATE_OBS datetime.datetime(2011, 8, 10, 22, 49, 22)and retrieve the image from the dataserver (download to your current working directory):
awe> raw.retrieve()once retrieved, you can display it:
awe> raw.display()or inspect it:
awe> raw.inspect()The last three commands are method calls. Methods typically perform an operation on a (Python) object. The object ``raw'' is called an instantation of the class RawTwilightFlatFrame. The last command brings up a graphical window: enter 'q' when you hover over it to quit the graphical window.
awe> chip = Chip() awe> chip.info() Chip: <astro.main.Chip.Chip object at 0xb03cd0ec> | +-name: +-object_id: '000000000000000000000000000000000
The Chip class has only one useful persistent attribute: name. object_id is a unique object identifier identifying each object in the database. Try to run the info method on all the other objects in these tutorials to get their information. Be aware, however, that not all objects have this method.
Notes:
This can be done in several ways. One way is to query for all instruments of which the name is not equal to an empty string. Alternatively you could use the ``like'' functionality to ask for all Instruments for which the name is equal to any string.
awe> q = Instrument.name != '' awe> for i in q: print(i.name) awe> q = Instrument.name.like('*') awe> for i in q: print(i.name)
Reminder: Instrument.get_persistent_properties() shows you that Instrument has the property name stored in the Astro-WISE database. Entering:
awe> Instrument.n
and hitting the Tab key also reveals that Instrument has the property name. NB: Tab completion does not work on queries (i.e., q.n following above example).
You can ask this of the context object, instantiated by default at the AWE prompt:
awe> context.get_chips_for_instrument('OMEGACAM')
awe> q = (RawScienceFrame.instrument.name == 'OMEGACAM') awe> len(q) 6592993
...and counting. Please note that for each OmegaCAM exposure 32 (one for each CCD) RawScienceFrames are created in the database.
awe> sci = ((ReducedScienceFrame.DATE_OBS == datetime.datetime(2011, 8, 6, 8, 5, 20)) & \ (ReducedScienceFrame.chip.name == 'ESO_CCD_#65')).min('creation_date')
For this ReducedScienceFrame, retrieve from the Astro-WISE dataserver the image itself and the bias image that was used. The FITS images will be saved to the current working directory.
awe> sci.retrieve() awe> sci.bias.retrieve()
The images, such as ReducedScienceFrames, which you and others create in Astro-WISE are stored on the Astro-WISE dataservers. As explained in more detail later, this will only be done if you explicitly commit these images to Astro-WISE. Even after committing them, you can invalidate images at any time if you decide they are no good after all.
(NB: enter following 2 lines as one line on the awe prompt.) awe> print(sci.DATE_OBS, sci.OBJECT, sci.instrument.name, sci.filter.name, sci.chip.name) # Or awe> print(sci.DATE_OBS) awe> print(sci.OBJECT) # etc.
awe> for frame in sci.bias.raw_bias_frames: print(frame.DATE_OBS)
And to print the equivalent modified Julian date:
awe> for frame in sci.bias.raw_bias_frames: print(frame.MJD_OBS) or by calculating from the DATE_OBS: awe> from common.util.utilities import datetime_to_mjd awe> for frame in sci.bias.raw_bias_frames: print(datetime_to_mjd(frame.DATE_OBS))
The ``from ... import ...'' statement loads a method to do the conversion. To find out what can be imported from a module, simply import the module and use the built-in dir() function:
awe> from common.util import utilities awe> dir(utilities)
To get help on the entire module, use the built-in help() function:
awe> help(utilities)
Note that the flat-field used in the reduction is a MasterFlatFrame which is this case was created from a master dome (DomeFlatFrame).
(NB: make sure to indent the 2nd line below:) awe> for frame in sci.flat.domeflat.raw_domeflat_frames: print(frame.imstat.median)
The property of the RawBiasFrame that contains its image statistics is called ``imstat''. This is an instance of the ``Imstat'' class, which itself has properties such as ``mean'', ``median'' and ``stdev''. We need to combine this query with a query on the name of the instrument of the RawBiasFrame.
NB: print following lines on a single awe prompt line. awe> q = (RawBiasFrame.instrument.name == 'OMEGACAM') & \ (RawBiasFrame.imstat.median > 1000.0) awe> len(q)
Here a lot of things come together. Usually, frame attributes that are all upper-case letters correspond to a FITS header keyword of the same name. This will apply to only some of our attributes. We will need to know the names of the other attributes that we are interested in, and if they can be queried on at all. This is done as follows:
awe> RawScienceFrame.get_persistent_properties()
In particular note the DATE_OBS, ``astrom'' and OBJECT properties. The R.A. and Dec can be found in ``astrom'' (as CRVAL1 and CRVAL2, respectively). In addition, we are directly querying on a datetime object (DATE_OBS) in order to get the data for 2011. This requires that you create a datetime object for your dates, so you can compare the two.
It may be helpful to limit the query to a single CCD, in order to avoid getting 32 times the same information (once for each CCD of the OmegaCAM camera).
awe> dat1 = datetime.datetime(2011,11,1) awe> dat2 = datetime.datetime(2011,12,1) awe> q = (RawScienceFrame.DATE_OBS>dat1) & \ (RawScienceFrame.DATE_OBS<dat2) & \ (RawScienceFrame.chip.name=='ESO_CCD_#65') & \ (RawScienceFrame.instrument.name=='OMEGACAM') awe> for s in q: print(s.DATE_OBS, s.filter.name, s.OBJECT, s.astrom.CRVAL1, s.astrom.CRVAL2)
awe> os.system('ls') awe> os.system('pwd')
awe> command = 'skycat file.fits&' awe> os.system(command)
To move one directory up use:
awe> os.chdir('..')
awe> RawScienceFrame.instrument.ThisAttributeDoesNotExist=='bla' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Software/users/astro-wise/awehome35/AWBASE/common/database/DBProperties.py", line 226, in __getattr__ persistent_property.__getattr__(self, attr) File "/Software/users/astro-wise/awehome35/AWBASE/common/database/DBProperties.py", line 140, in __getattr__ self.attribute, self.cls.__name__, attr)) AttributeError: Persistent attribute "instrument" of class "RawScienceFrame" does not have a persistent attribute "ThisAttributeDoesNotExist"
/portal/howtos/man_howto_tutorial_introduction/man_howto_tutorial_introduction.shtmlprevious | /portal/howtos/man_howto_tutorial_introduction/man_howto_tutorial_introduction.shtmlhome | /portal/howtos/man_howto_tutorial_science/man_howto_tutorial_science.shtmlnext |