Integration of data reduction and near real-time archiving into the Keck Observing Model

Max N. Brodheim; John M. O'Meara; Jeffrey A. Mader; G. Bruce Berriman; Matthew K. Brown; Lucas Fuhrman; Tyler Tucker; Christopher R. Gelino; Meca S. Lynn; Melanie A. Swain

doi:10.1117/12.2629551

25 August 2022 Integration of data reduction and near real-time archiving into the Keck Observing Model

Max N. Brodheim, John M. O'Meara, Jeffrey A. Mader, G. Bruce Berriman, Matthew K. Brown, Lucas Fuhrman, Tyler Tucker, Christopher R. Gelino, Meca S. Lynn, Melanie A. Swain

Author Affiliations +

Proceedings Volume 12186, Observatory Operations: Strategies, Processes, and Systems IX; 121860H (2022) https://doi.org/10.1117/12.2629551
Event: SPIE Astronomical Telescopes + Instrumentation, 2022, Montréal, Québec, Canada

Abstract

The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory’s Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to Keck’s internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections. The Keck Cosmic Web Imager (KCWI) instrument is the first whose data are managed this way. It showcases how KOA integrates into an observing night, provides the data needed to make real-time adjustments to observing, and delivers products that allow for faster publication by both our observers and archival researchers. This effort has involved the delivery of new, compact, Python-based data preparation and ingestion software. We also discuss the new and updated Data Reduction Pipelines (DRPs) required to generate science-ready data, how their development and deployment enables the delivery of these products, and how Keck’s commitment to maintaining DRPs in-house will result in more robust datasets for all our observers and KOA users.

Conference Presentation

Citation Download Citation

Max N. Brodheim, John M. O'Meara, Jeffrey A. Mader, G. Bruce Berriman, Matthew K. Brown, Lucas Fuhrman, Tyler Tucker, Christopher R. Gelino, Meca S. Lynn, and Melanie A. Swain "Integration of data reduction and near real-time archiving into the Keck Observing Model", Proc. SPIE 12186, Observatory Operations: Strategies, Processes, and Systems IX, 121860H (25 August 2022); https://doi.org/10.1117/12.2629551

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available