Astro-Wise Federation

i. Dataservers

handle all files - 99% being image data
dataservers know about each other
files are store()d on one local dataserver
files are retrieve()d from the local dataserver
if a file is not available locally it is retrieve()d automatically from one of the other dataservers
files that were not locally available are cached to speed up future access
dataservers are implemented in 100% Python
dataservers use HTTP protocol including referrals

files can be accessed by URL with any browser or wget (curl).

Current situation

single dataservers in use at the different Astro-Wise sites
105,385 files at OmegaCEN

Soon

multiple dataservers in use at OmegaCEN
dataservers "connected" between Astro-Wise sites

ii. CVS

configuration files + source code are handled by CVS
central repository
local copies / check outs
STABLE version

Current situation

in use at all Astro-Wise sites - more than 40,000 changes to odoco, opipe and awhtml.
automatic updates

iii. Database federation

persistent objects are distributed by copying
- through Oracle Streams
- using Python layer to access remote tables via database links
Copy objects, including identity
Copy REFerences
Oracle Streams does not support user-defined types
valid for all DBObjects i.e.
- CalFiles
- METAdata of Raw and (Reduced)ScienceFrames and CalibrationFrames - such as image statistics

Current situation

multi-user databases operational at the Astro-Wise sites
Python based table copy being implemented using object identity

Soon

federation-wide users
per user / context read/write and delete permission

Persistency Basics

Python class definition also creates DDL
Oracle User-Defined Types are similar to classes
Includes inheritance and methods (PL/SQL, C/C++, Java) and references to objects
Python classes and attributes are made persistent by inheriting from a persistent base class
Corresponding Oracle TYPE, TABLE and VIEW are created, if absent
Object TABLE containing objects of exactly one TYPE
Object VIEW contains objects of the TYPE and all its subtypes
A `link' attribute refers to an object. In Oracle a REF is used
Instances are made persistent with .commit()
.commit() works recursively
Objects can be retrieved by object_id or with a Python query
similar to Conceptual Queries
Python queries are Select objects
- A Select object can be a logical combination of Select objects
- Select objects are transformed into SQL not earlier than necessary
- Select object provides an iterator that returns Python objects

class BiasFrame(BaseFrame):
    instrument      = persistent('', Instrument, None)
    chip            = persistent('', Chip, None)
    observing_block = persistent('', ObservingBlock, None)
    timestamp_start = persistent('', DateTime.DateTimeType, DateTime.DateTime(1990,1,1))
    timestamp_end   = persistent('', DateTime.DateTimeType, DateTime.DateTime(2030,1,1))
    creation_date   = persistent('', DateTime.DateTimeType, DateTime.DateTime(1990,1,1))
    raw_bias_frames = persistent('The frames used to construct this frame', RawBiasFrame, [])
    process_params = persistent('Processing parameters', BiasFrameParameters, None)
    prev            = persistent('Comparison frame')

CREATE TYPE "BiasFrame$"
UNDER "BaseFrame$"
(
   "raw_bias_frames" "BiasFrame$raw_bias_frames",
   "instrument" REF "Instrument$",
   "timestamp_start" DATE,
   "prev" REF "BiasFrame$",
   "observing_block" REF "ObservingBlock$",
   "chip" REF "Chip$","timestamp_end" DATE,
   "process_params" REF "BiasFrameParameters$",
   "creation_date" DATE
)
NOT FINAL
;

CREATE TABLE "BiasFrame"
OF "BiasFrame$"
NOT SUBSTITUTABLE AT ALL LEVELS
NESTED TABLE "raw_bias_frames"
STORE AS "BiasFrame-raw_bias_frames"
;

CREATE VIEW "BiasFrame+"
OF "BiasFrame$"
UNDER "BaseFrame+"
AS
   SELECT *
   FROM "BiasFrame"
   WITH READ ONLY
;

for flat in (TwilightFlatFrame.chip.name == 'A5506-4') &
            (TwilightFlatFrame.creation_date > DateTime.DateTime(2003,11,31)):

    print flat.filename, flat.bias.filename

    flat.retrieve()
    flat.bias.retrieve()

Scalability

Partitioned tables and indexes

Logically and physically divide large tables into partitions/tablespaces
Tables and indexes can be partitioned independently
Range, hash, list partitioning
Partitioning can be changed in a running database
Nested tables cannot be partitioned