WP3: Provide Federated database

Objectives:
  • Design and implement federated databases that distribute documentation, source code, tools, pipeline administration, and calibration data over the participating sites as a support tool for WP1 and WP2
This wp provides the project management's backbone of the survey system.

Work description:

  • define classes/standards
  • select data base engines
  • define interfaces to pipeline and tools environment /implement
  • define query language/method /implement
  • define scripting /implement
  • implement classes interfaces to data bases

Planned total man month over 4 years (from contract)
 

NOVA   ESO   OAC   TERAPIX   USM  
Boxhoorn 30 Slijkhuis
 6
 Getman 9 Bertin 2 EC-Snigula 3
Rengelink  4 EIS - NN 6 Puddu 3 Domisse 8    
EC-NN 12 EC-NN 6 EC-Volpicelli 6 EC-Magnard 6    
Valentijn  6 Rite 6     Erben- Didelon 17    
Total 52   24
   18   33   3
WP 3 130              

NB 6 month of Rite moved from WP2 to 3 


TASK LEADER Boxhoorn + Valentijn

Current Group Domisse, Getman, Rite, Slijkhuis, Snigula

MILESTONES
Phase I: be ready to receive first data without later re-definition or re-shuffling - 2002
  1. T0 kick-off- prepare design review for WP 3
  2. T0 + 1Q design review WP3
Phase II: evaluate and prepare for mass production -2003 -2004
  1. T0 + 7Q WP3: One year operations and population of data base review. deliverables:  populated calibration database
  1. T0 + 10Q System full mass production proof (i.e. 100 Tbyte data volume) deliverable:  data results in Tbyte regime
Phase III: qualify, operate and populate the mass production system -2005
  1. T0 + 11Q Review and set goals of final delivery system
  2. T0 + 15Q End of implementation/development work of this programme. Deliverable- qualified database for Survey System
Requirements for AstroWISE db
  1. Administrate all image pipeline and calibration pipeline operation at all partners data centres
  2. Provide workhorse for queries/searches in multi-Terabyte sourcelists
  3. Distribution of all the data base contents over the consortium with an operator tuning of assigning trans-site pointers (a la URL) versus replication
  4. database functions both for end-users (astronomers) and infra-structure operators
Subitems from requirements: (what Oracle gives/ 17 May 2002 perception )


 Comparison of Oracle, MySQL and Sybase for various items of interest for ASTRO-WISE
Subitem Oracle MySQL SyBase
Support OO coding / persistency. OK / 9i feature. NO / Absent. NO / Absent.
DB both suitable for pipeline administration, advanced applications and sourcelists OK / see also persistency Appendix . OK for pipeline administration / can be done with SQL92.
NO for sourcelists and advanced applications / due to lack of OO functionality.		 OK for pipeline administration / can be done with SQL92.
NO for sourcelists and advanced applications / due to lack of OO functionality.
Tbyte scalability of sourcelist (table) data. OK / see internal report . NO / see MySQL documentation . OK / see SyBase article .
Support tools like Sassociate. OK / see internal report - under investigation at Oracle HQ. NO / Absent. OK? / via proprietary third party addon named SQS .
Provide components for creating federation. OK / advanced replication - peer to peer Meta data. NO / Absent. UNKNOWN / Use SyBase Replication Server+Agent? Peer-to-peer support unknown.
Provide tools/infrastructure for DB management. OK / Oracle per se . NO / via command line tool. OK? / via SyBase Central and SQL Remote. Seems to be MS-Win only. Linux?
Accessibility by programmer/user and training. OK / Python-SQL interface, training for local juniors, central DBA. OK / Python-SQL interface, OpenSource community. NO / Python-SQL interface.
Design support. OK / arranged. NO / external consultancy? NO / subject to availability and cost.
License costs. OK / arranged for national centers, few 100 euro for individuals. OK / Free of charge!!! ? / See list price .

ACTIONS Q1 2002 - all closed
  • ASTRO-WISE contract with Oracle has been signed  27/5/2002

ACTIONS Q2 2002  (after 17 May meeting Paris) / status as of 1/11/02
  1. Make plan for db development/implementation  - federated design with John Abrahams (Oracle)  
    1. 1/11/02: prototype available now; JA wil visit Groningen  26/11/02
  2. go for detailed design - 1/11/02: well advanced 
  3. set up course Oracle (also Python and pipeline)  dedicated for ASTRO-WISE personnel and student
tentatively planned during the week 7 -12 Oktober 2002 Lorentz center Leiden (?) (done - closed 1/11/03)

ACTIONS Q3 2002 (after team meeting 31/10/02)
  1.  USM with the assistance of all: A database with reference data, like transmissions of filters etc will have to be set up. Bertin will provide some references.