LSST Data Management Applications UML Use Case and Activity Model LDM-1348/18/2011

Large Synoptic Survey Telescope (LSST)

Data Management Applications UML Use Case Model

Mario Juric, Robyn Allsman, Jeff Kantor

LDM-134

Latest Revision: October10, 2013

This LSST document has been approved as a Content-Controlled Document by the LSST DM Technical Control Team. If this document is changed or superseded, the new document will retain the Handle designation shown above. The control is on the most recent digital document with this Handle in the LSST digital archive and not printed versions. Additional information may be found in the LSST DM TCT minutes.

The contents of this document are subject to configuration control by the LSST DM Technical Control Team.

1

LSST Data Management Applications UML Use Case and Activity Model LDM-134 10/10/2013

Change Record

Version / Date / Description / Owner name
1 / 1/28/2011 / Update Document to reflect Model based on Data Challenge 3 / J. Kantor
2 / 7/12/2011 / Update Document to reflect Model based on Data Challenge 3B PT1 / R. Allsman
3 / 8/18/2011 / General updates / R. Allsman
4 / 9/11/2013 / Final Design updates (revision 1.125) / R.Allsman
5 / 9/26/2013 / Formatting updates and WBS inclusion / R. Allsman
6 / 10/10/2013 / Returned SDQA to the Apps model (revision 1.144); TCT approved / R. Allsman

Table of Contents

Change Record

DMS Use Cases

Actors

Science Data Calibration and Quality Assessment

Science Data Quality Assessment Pipeline

Assess Data Quality

Assess Data Quality for Calibration Products

Assess Data Quality for Nightly Processing at Archive

Assess Data Quality for Data Release

Assess Data Quality for Nightly Processing

Science Data Quality Analyst Toolkit

Analyze SDQA Metrics

Correlate SDQA metric with other data

Correlate SDQA metrics

Display SDQA Metrics

Science Pipeline Toolkit

Science Pipeline Toolkit

Configure Pipeline Execution

Execute Pipeline

Incorporate User Code into Pipeline

Monitor Pipeline Execution

Select Data to be Processed

Select Data to be Stored

Upload User Codes

Calibration Processing

Periodic Calibration Products Production

Produce Calibration Data Products

Acquire Raw Calibration Exposures

Calculate System Bandpasses

Calculate Telescope Bandpasses

Construct Defect Map

Produce Crosstalk Correction Matrix

Produce Master Bias Exposure

Produce Master Dark Exposure

Produce Master Fringe Exposures

Produce Master Pupil Ghost Exposure

Produce Optical Ghost Catalog

Produce Synthetic Flat Exposures

Determine Illumination Correction

Produce Master Flat-Spectrum Flat Exposures

Correct Monochromatic Flats

Create Master Flat-Spectrum Flat

Create Master Illumination Correction

Determine CCOB-derived Illumination Correction

Determine Optical Model-derived Illumination Correction

Determine Self-calibration Correction-Derived Illumination Correction

Determine Star Raster Photometry-derived Illumination Correction

Nightly Calibration Products

Calculate Atmospheric Models from Calibration Telescope Spectra

Prepare Nightly Flat Exposures

Reduce Spectrum Exposure

Common Image Processing

Low-level Image Operations

Raw Exposure Processing

Calibrate Exposure

Combine Raw Exposures

Process Raw Exposures to Calibrated Exposure

Remove Instrument Signature

Assemble CCD

Detect Sources

Determine Aperture Correction

Determine Photometric Zeropoint

Determine PSF

Determine Sky Background Model

Determine WCS

Remove Exposure Artifacts

Sum Exposures

Nightly Processing

Prepare for Observing

Process Nightly Observing Run

Association

Perform DIA Source Association

Perform DIA Object Association

Create Instance Catalog for Visit

Associate with Instance Catalog

Alert Generation and Distribution

Generate and Distribute Alerts

Generate Alerts

Distribute to Subscribed Brokers

Distribute to Subscribed Users

DIA Source Detection and Characterization

Detect and Characterize DIA Sources

Estimate Detection Efficiency

Subtract Calibrated Exposure from Template Exposure

Detect DIA Sources in Difference Exposure

Measure DIA Sources

Measure Snap Difference Flux

Identify DIA Sources caused by Artifacts

Perform Difference Image Forced Photometry

Perform Precovery Forced Photometry

DIA Object Characterization

Update DIA Object Properties

Calculate DIA Object Flux Variability Metrics

Fit DIA Object Position and Motion

Moving Objects Processing

Process Moving Objects

Find Tracklets

Link Tracklets into Tracks

Fit Orbit

Prune Moving Object Catalog

Perform Precovery

Recalculate Solar System Object Properties

Data Release Processing

Perform Global Self-Calibration

Produce a Data Release

Cross-match Previous Release AstroObject IDs

Global Photometric Calibration

Perform Global Photometric Calibration

Global Astrometric Calibration

Perform Global Astrometric Calibration

Single Visit Processing

Perform Single Visit Processing

Measure Single Visit Sources

PSF Estimation

Perform Full Focal Plane PSF Estimation

Difference Image Characterization

Detect and Characterize DIA Objects

Deep Detection

Detect and Characterize AstroObjects

Detect Sources on Coadds

Image Coaddition

Create Template Exposures

Create Coadd Exposures

Coadd Calibrated Exposures

Create Deep Coadd Exposures

Create Short Period Coadd Exposures

Create Best Seeing Coadd Exposures

Create PSF-matched Coadd Exposures

Object Characterization

Characterize AstroObject Flux Variability

Create Sky Coverage Maps

Measure AstroObjects

Perform Deblending and Association

Perform Forced Photometry

The contents of this document are subject to configuration control by the LSST DM Technical Control Team.

1

LSST Data Management Applications UML Use Case and Activity Model LDM-134 10/10/2013

Model Documentation

Section: Section1

DMS Use Cases

WBS:: 02C

The DMS Use Case Model captures the conceptual definition and relationships of the DMS processing elements. It is semantically very close in level to the OSS and DMSR.

The Use Case Model is described in the form of Unified Modeling Language (UML) 2.0. There are two types of diagrams included:

Package diagrams - Show the overall grouping of model elements into topical areas or modeling packages.

Use Case Diagrams - Show the interactions between human users and external systems (actors) that interact with the DMS. Also show the main processes (use cases) that occur within the DMS during operation of the system in response to these interactions.

The elements on the diagrams are each further defined in structured text. This text describes how the processing creates, updates, uses, and/or destroys Domain Classes. In certain cases, a Use Case may "invoke" (perform in-line) another Use Case. Sequencing of the structured text allows branching from and rejoining to the basic path based on specific criteria.

Figure 1 : DMS Use Cases Packages

This diagram depicts the packages contained in the DMS Use Case Model.

Actors

WBS:: 02C

This package contains the DMS Actors, which are human users of the DMS and external systems with which the DMS interacts.

Figure 2 : Actors

Actor / Description
Public Interface User / This actor represents all users/systems that access LSST public interfaces
Public Resource Locator / This is an external system that contains locations and/or access information to public astronomical resources, such as surveys, tools, and services.
Pipeline Creator / This actor is any user that has the access necessary to create a new component or pipeline type or a new instance of an existing component or pipeline type and to cause that instance to be available for execution.
Pipeline Operator / This actor is any user with access to cause pipelines to execute, to terminate, or to be stopped and started.
Science User / This actor is any user who has access to LSST Data Products, Pipelines, or both.
Simulator / This actor represents any source of simlulated LSST science data, including images, meta-data, catalog data, alerts, etc.
Telescope / This is the main LSST Observatory Telescope.
Observatory Operations / This actor has authority to permit LSST Data Products to be released external to the project.
Camera / This actor represents the Camera subsystem of the LSST, including the Science Data Subsystem (SDS) which is the primary Camera interface to the DMS.
Catalog Creator / This actor is any user that has the access necessary to create a new catalog type or a new instance of an existing catalog type and to cause that instance to be populated with data.
Alert Category Author / This is a user that sets up LSST Alert Categories, allowing for later Subscriptions to these Categories
Auxiliary Telescope / This is the auxiliary telescope used for calibration.
Data Management System Administrator / This actor is any user that has the access necessary to invoke system administration operations (e.g. configure security, equipment, system parameters, etc.) in the LSST Data Management Control System.
LSST Operations / This actor is any user that performs an operational role in the LSST Observatory, including operators and administrators.
Observatory Control System / This actor represents the overall master control system that coordinates the operation of all LSST subsystems.
DMS User / This actor is any user that can access the DMS in any manner. It is the most general class of user, and therefore the least privileged.
DMS-External System / This is any system not part of the DMS with which the DMS has an interface.

Science Data Calibration and Quality Assessment

WBS:: 02C.01.02

Science Data Calibration and Quality Assessment includes

- Science Data Quality Assessment pipelines and toolkits for use by data analysts and scientists to assess the quality of the DMS-generated data;

- Science Data Pipeline Toolkit; and

- Calibration Products Pipeline.

Science Data Quality Assessment Pipeline

WBS:: 02C.01.02.01

Science Data Quality Assessment Pipeline implements the SDQA Pipeline capabilities.

Figure 3 : Science Data Quality Assessment Pipeline

Figure 4 : Assess Data Quality

Assess Data Quality

WBS: 02C.01.02.01

Assess Data Quality allows the user to choose the data type to be examined and assessed.

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. When ( user selects Calibration Products ):(see AltPath: User Selects Data Release)(see AltPath: User selects Nightly Processing at Base)(see AltPath: User selects Nightly Processing at Archive)
  2. invoke: Assess Calibration Products
  3. fin:

AltPath: User Selects Data Release /
  1. invoke: Assess Data Quality for Calibration Products
/ Basic Path step:3
AltPath: User selects Nightly Processing at Archive /
  1. invoke: Assess Data Quality for Nightly Processing at Archive
/ Basic Path step:3
AltPath: User selects Nightly Processing at Base /
  1. invoke: Assess Data Quality for Nightly Processing at Base
/ Basic Path step:3
Assess Data Quality for Calibration Products

WBS: 02C.01.02.01

Assess Data Qualityfor Calibration Products

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Assess Data Quality for Nightly Processing at Archive

WBS: 02C.01.02.01

Assess Data Quality for Nightly Processing at Archive - On completion of a pre-defined number of observing nights or on command by Observatory Operations, the DMS does a complete assessment of the overall state of the LSST Data Products and produces Data Product Quality Reports. This assessment looks at the SRD-required observatory and mission satisfaction metrics, such as fields visited in each filter, % of raw images within photometric/astrometric specifications, etc.

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. Do Analyze Image Quality:
  2. ....Quality of calibration steps (flatfield, debias, defringe, etc); Flag Outliers
  3. ....Analyse artifacts: cosmic rays; ccd traps, bad columns, etc; satellite trails; stray light; Flag Outliers
  4. ....Analyze telescope optical performance: psf shape over the field and associated wavefront params; Flag Outliers
  5. ....Determine atmospheric seeing parameters - including spatial correlation; Flag Outliers
  6. done
  7. Do Analyze Photometric Qualtiy using several methods:
  8. ....Lightcurve analysis
  9. ....CMD analysis
  10. ....Global consistency of standards; Flag Outliers
  11. done:
  12. Do AnalyzeAstrometric Quality:
  13. ....Analyzeastrometric solutions in the image WCS; Flag outliers
  14. ....Analyze proper motion/parallax solutions in the object database; Flag outliers
  15. done:
  16. Do Analyze Orbit Quality:
  17. ....Analyze quality of fit of orbits to observations; Flag Outliers
  18. done:
  19. Do Analyze Object Properties' Quality:
  20. ....Shape; Flag Outliers
  21. ....Type classification; Flag Outliers
  22. ....Deblending; Flag Outliers
  23. ....Photo Z; Flag Outliers
  24. done:
  25. Analyze Outliers

Assess Data Quality for Data Release

WBS: 02C.01.02.01

Assess Data Qualityfor Data Release

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Assess Data Quality for Nightly Processing

WBS: 02C.01.02.01

Assess Data Qualityfor Nightly Processing

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Science Data Quality Analyst Toolkit

WBS:: 02C.01.02.02

Figure 5 : Science Data Quality Analysis

Analyze SDQA Metrics

WBS: 02C.01.02.02

Analyze SDQA Metrics - when provided with selection and output criteria, the system acquires and formats the data according to the users' output preference.

GIVEN:

Analyst is using web-base tool to perform these tasks

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. System displays the Task Selection Page.
  2. SDQA Analyst selects the "Analyze SDQA Metrics".
  3. When (SDQA Metrcis selections exists):(see AltPath: SDQA Metric not populated yet)
  4. System displays the SDQA Metric Set Up Page.
  5. SDQA Analyst selects the SDQA Metrics, Sky Region, Focal Plane Region, Time Range, and output format.
  6. System queries the SDQA Data Archive,
  7. System generates SDQA Results,
  8. System formats them for output, and displays them on the SDQA Results Page.
  9. fin:

AltPath: SDQA Metric not populated yet /
  1. System generates partial SDQA Results and displays warning that not all SDQA Results are available.
/ Basic Path step:1
Correlate SDQA metric with other data

WBS: 02C.01.02.02

Correlate SDQA metric with other data - e.g. seeing from ancillary telescope, voltages from facilities database, number of sources extracted for a given exposure from DM database, etc.

GIVEN:

Analyst is using web-base tool to perform these tasks

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Correlate SDQA metrics

WBS: 02C.01.02.02

Correlate SDQA Metrics - with each other: e.g. noise level with number of poor extractions.

GIVEN:

Analyst is using web-base tool to perform these tasks

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Display SDQA Metrics

WBS: 02C.01.02.02

Display SDQA Metrics - when provided with the constraints of the data-of-interest and graphics style.

GIVEN:

Analyst is using web-base tool to perform these tasks

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. Select metric of interest (e.g. DC3 SDQA metric) .
  2. Select region of focal plane (e.g. segment, CCD, raft, or full focal plane).
  3. Select time range (e.g. full night, one exposure, etc.).
  4. Select display type (histogram, xy plot, image).
  5. Display chosen metric in chosen style.

Science Pipeline Toolkit

WBS:: 02C.01.02.03

Science Pipeline Toolkit provides capabilities that permit users of the DMS to perform processing of LSST data with LSST open software and user-supplied codes, for Level 3 Data Product production.

Figure 6 : Science Pipeline Toolkit

Science Pipeline Toolkit

WBS: 02C.01.02.03

Science Pipeline Toolkit provides capabilities that permit users of the DMS to perform processing of LSST data with LSST open software and user-supplied codes, for Level 3 Data Product production.

Given:

Science Pipeline Toolkit is a controller providingthe following capabilities:

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. When ( upload requested ):(see AltPath: User Code Incorporation Requested)(see AltPath: Data is selected to be Stored)(see AltPath: Pipeline is selected to be Executed)(see AltPath: Pipeline Execution is selected to be Monitored)(see AltPath: Pipeline Execution is selected to be Configured)(see AltPath: Data to be Processed is selected)
  2. upload user codes

AltPath: Data is selected to be Processed /
  1. invoke: Select Data to be Processsed

AltPath: Data is selected to be Stored /
  1. invoke: Select Data to be Stored
/ Basic Path step:1
AltPath: Data to be Processed is selected /
  1. invoke: Select Data to be Processed
/ Basic Path step:1
AltPath: Pipeline Execution is selected to be Configured /
  1. invoke: Configure Pipeline Execution
/ Basic Path step:1
AltPath: Pipeline Execution is selected to be Monitored /
  1. invoke: Monitor Pipeline Execution
/ Basic Path step:1
AltPath: Pipeline is selected to be Executed /
  1. invoke: Execute Pipeline
/ Basic Path step:1
AltPath: User Code Incorporation Requested /
  1. invoke: incorporate User Code into Pipeline
/ Basic Path step:1
Configure Pipeline Execution

WBS: 02C.01.02.03

Configure Pipeline Execution

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Execute Pipeline

WBS: 02C.01.02.03

Execute Pipeline

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Incorporate User Code into Pipeline

WBS: 02C.01.02.03

Incorporate User Code into Pipeline

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Monitor Pipeline Execution

WBS: 02C.01.02.03

Monitor Pipeline Execution

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Select Data to be Processed

WBS: 02C.01.02.03

Select Data to Process

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Select Data to be Stored

WBS: 02C.01.02.03

Select Data to be Stored

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Upload User Codes

WBS: 02C.01.02.03

Upload User Code

Scenario / Steps Summary / Rejoins at
Basic Path /
  1. TBD

Calibration Processing

WBS:: 02C.01.02.04

Calibration Products Processing:

- generates the Calibration Products used to remove the instrument signature from Raw Exposures;

- generates the Calibration Data based on Engineering and Facility Database and Auxiliary Telescope.

Periodic Calibration Products Production

WBS:: 02C.01.02.04

Figure 7 : Produce Calibration Data Products

Produce Calibration Data Products

WBS: 02C.01.02.04