Automated data reduction workflows for astronomy

The ESO Reflex environment

W. Freudling, M. Romaniello, D. M. Bramich, P. Ballester, V. Forchi, C. E. García-Dabló, S. Moehler, M. J. Neeser

Research output: Contribution to journalArticle

59 Citations (Scopus)

Abstract

Context. Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. Aims. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute a sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitive representation, execution and modification of the data reduction workflow, and has facilities for inspection and interaction with the data. Methods. The European Southern Observatory (ESO) has developed Reflex, an environment to automate data reduction workflows. Reflex is implemented as a package of customized components for the Kepler workflow engine. Kepler provides the graphical user interface to create an executable flowchart-like representation of the data reduction process. Key features of Reflex are a rule-based data organiser, infrastructure to re-use results, thorough book-keeping, data progeny tracking, interactive user interfaces, and a novel concept to exploit information created during data organisation for the workflow execution. Results. Automated workflows can greatly increase the efficiency of astronomical data reduction. In Reflex, workflows can be run non-interactively as a first step. Subsequent optimization can then be carried out while transparently re-using all unchanged intermediate products. We found that such workflows enable the reduction of complex data by non-expert users and minimizes mistakes due to book-keeping errors. Conclusions. Reflex includes novel concepts to increase the efficiency of astronomical data processing. While Reflex is a specific implementation of astronomical scientific workflows within the Kepler workflow engine, the overall design choices and methods can also be applied to other environments for running automated science workflows.

Original languageEnglish
Article numberA96
JournalAstronomy and Astrophysics
Volume559
DOIs
Publication statusPublished - 1 Nov 2013
Externally publishedYes

Fingerprint

reflexes
data reduction
astronomy
European Southern Observatory
observatory
engines
progeny
graphical user interface
software development tools
reuse
files
inspection
engine
optimization
products

Keywords

  • Astronomical databases: miscellaneous
  • Methods: data analysis
  • Techniques: miscellaneous
  • Virtual observatory tools

ASJC Scopus subject areas

  • Astronomy and Astrophysics
  • Space and Planetary Science

Cite this

Freudling, W., Romaniello, M., Bramich, D. M., Ballester, P., Forchi, V., García-Dabló, C. E., ... Neeser, M. J. (2013). Automated data reduction workflows for astronomy: The ESO Reflex environment. Astronomy and Astrophysics, 559, [A96]. https://doi.org/10.1051/0004-6361/201322494

Automated data reduction workflows for astronomy : The ESO Reflex environment. / Freudling, W.; Romaniello, M.; Bramich, D. M.; Ballester, P.; Forchi, V.; García-Dabló, C. E.; Moehler, S.; Neeser, M. J.

In: Astronomy and Astrophysics, Vol. 559, A96, 01.11.2013.

Research output: Contribution to journalArticle

Freudling, W, Romaniello, M, Bramich, DM, Ballester, P, Forchi, V, García-Dabló, CE, Moehler, S & Neeser, MJ 2013, 'Automated data reduction workflows for astronomy: The ESO Reflex environment', Astronomy and Astrophysics, vol. 559, A96. https://doi.org/10.1051/0004-6361/201322494
Freudling, W. ; Romaniello, M. ; Bramich, D. M. ; Ballester, P. ; Forchi, V. ; García-Dabló, C. E. ; Moehler, S. ; Neeser, M. J. / Automated data reduction workflows for astronomy : The ESO Reflex environment. In: Astronomy and Astrophysics. 2013 ; Vol. 559.
@article{7ea032c1b4f448c9bd1a97bb482a2cb1,
title = "Automated data reduction workflows for astronomy: The ESO Reflex environment",
abstract = "Context. Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. Aims. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute a sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitive representation, execution and modification of the data reduction workflow, and has facilities for inspection and interaction with the data. Methods. The European Southern Observatory (ESO) has developed Reflex, an environment to automate data reduction workflows. Reflex is implemented as a package of customized components for the Kepler workflow engine. Kepler provides the graphical user interface to create an executable flowchart-like representation of the data reduction process. Key features of Reflex are a rule-based data organiser, infrastructure to re-use results, thorough book-keeping, data progeny tracking, interactive user interfaces, and a novel concept to exploit information created during data organisation for the workflow execution. Results. Automated workflows can greatly increase the efficiency of astronomical data reduction. In Reflex, workflows can be run non-interactively as a first step. Subsequent optimization can then be carried out while transparently re-using all unchanged intermediate products. We found that such workflows enable the reduction of complex data by non-expert users and minimizes mistakes due to book-keeping errors. Conclusions. Reflex includes novel concepts to increase the efficiency of astronomical data processing. While Reflex is a specific implementation of astronomical scientific workflows within the Kepler workflow engine, the overall design choices and methods can also be applied to other environments for running automated science workflows.",
keywords = "Astronomical databases: miscellaneous, Methods: data analysis, Techniques: miscellaneous, Virtual observatory tools",
author = "W. Freudling and M. Romaniello and Bramich, {D. M.} and P. Ballester and V. Forchi and Garc{\'i}a-Dabl{\'o}, {C. E.} and S. Moehler and Neeser, {M. J.}",
year = "2013",
month = "11",
day = "1",
doi = "10.1051/0004-6361/201322494",
language = "English",
volume = "559",
journal = "Astronomy and Astrophysics",
issn = "0004-6361",
publisher = "EDP Sciences",

}

TY - JOUR

T1 - Automated data reduction workflows for astronomy

T2 - The ESO Reflex environment

AU - Freudling, W.

AU - Romaniello, M.

AU - Bramich, D. M.

AU - Ballester, P.

AU - Forchi, V.

AU - García-Dabló, C. E.

AU - Moehler, S.

AU - Neeser, M. J.

PY - 2013/11/1

Y1 - 2013/11/1

N2 - Context. Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. Aims. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute a sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitive representation, execution and modification of the data reduction workflow, and has facilities for inspection and interaction with the data. Methods. The European Southern Observatory (ESO) has developed Reflex, an environment to automate data reduction workflows. Reflex is implemented as a package of customized components for the Kepler workflow engine. Kepler provides the graphical user interface to create an executable flowchart-like representation of the data reduction process. Key features of Reflex are a rule-based data organiser, infrastructure to re-use results, thorough book-keeping, data progeny tracking, interactive user interfaces, and a novel concept to exploit information created during data organisation for the workflow execution. Results. Automated workflows can greatly increase the efficiency of astronomical data reduction. In Reflex, workflows can be run non-interactively as a first step. Subsequent optimization can then be carried out while transparently re-using all unchanged intermediate products. We found that such workflows enable the reduction of complex data by non-expert users and minimizes mistakes due to book-keeping errors. Conclusions. Reflex includes novel concepts to increase the efficiency of astronomical data processing. While Reflex is a specific implementation of astronomical scientific workflows within the Kepler workflow engine, the overall design choices and methods can also be applied to other environments for running automated science workflows.

AB - Context. Data from complex modern astronomical instruments often consist of a large number of different science and calibration files, and their reduction requires a variety of software tools. The execution chain of the tools represents a complex workflow that needs to be tuned and supervised, often by individual researchers that are not necessarily experts for any specific instrument. Aims. The efficiency of data reduction can be improved by using automatic workflows to organise data and execute a sequence of data reduction steps. To realize such efficiency gains, we designed a system that allows intuitive representation, execution and modification of the data reduction workflow, and has facilities for inspection and interaction with the data. Methods. The European Southern Observatory (ESO) has developed Reflex, an environment to automate data reduction workflows. Reflex is implemented as a package of customized components for the Kepler workflow engine. Kepler provides the graphical user interface to create an executable flowchart-like representation of the data reduction process. Key features of Reflex are a rule-based data organiser, infrastructure to re-use results, thorough book-keeping, data progeny tracking, interactive user interfaces, and a novel concept to exploit information created during data organisation for the workflow execution. Results. Automated workflows can greatly increase the efficiency of astronomical data reduction. In Reflex, workflows can be run non-interactively as a first step. Subsequent optimization can then be carried out while transparently re-using all unchanged intermediate products. We found that such workflows enable the reduction of complex data by non-expert users and minimizes mistakes due to book-keeping errors. Conclusions. Reflex includes novel concepts to increase the efficiency of astronomical data processing. While Reflex is a specific implementation of astronomical scientific workflows within the Kepler workflow engine, the overall design choices and methods can also be applied to other environments for running automated science workflows.

KW - Astronomical databases: miscellaneous

KW - Methods: data analysis

KW - Techniques: miscellaneous

KW - Virtual observatory tools

UR - http://www.scopus.com/inward/record.url?scp=84889001050&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84889001050&partnerID=8YFLogxK

U2 - 10.1051/0004-6361/201322494

DO - 10.1051/0004-6361/201322494

M3 - Article

VL - 559

JO - Astronomy and Astrophysics

JF - Astronomy and Astrophysics

SN - 0004-6361

M1 - A96

ER -