Datasink: an efficient open-source system for reliable transfers of observatory data and jobs

Eric Jeschke; Takeshi Inagaki; Russell Kackley

doi:10.1117/12.3019063

25 July 2024 Datasink: an efficient open-source system for reliable transfers of observatory data and jobs

Eric Jeschke, Takeshi Inagaki, Russell Kackley

Proceedings Volume 13101, Software and Cyberinfrastructure for Astronomy VIII; 131013E (2024) https://doi.org/10.1117/12.3019063
Event: SPIE Astronomical Telescopes + Instrumentation, 2024, Yokohama, Japan

Conference Poster

Abstract

We present a simple sytem for job distribution built on the RabbitMQ open-source message broker. The system is based on the concept of job sources (origins), sinks (destinations) and realms (hubs), where a network of these entities can be readily established with a configuration file for each site and a RabbitMQ server running at each hub. Jobs are sent via persistent JSON-encoded packets and delivered reliably by RabbitMQ queues. The system was built primarily for robust data tranfers amidst volatile network connections but is general enough for any kind of flexible job distribution scheme where reliable delivery of job messages is needed. We are releasing the ”datasink” as an open-source Python package on Github. Aside from RabbitMQ, there are minimal additional requirements.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Eric Jeschke, Takeshi Inagaki, and Russell Kackley "Datasink: an efficient open-source system for reliable transfers of observatory data and jobs", Proc. SPIE 13101, Software and Cyberinfrastructure for Astronomy VIII, 131013E (25 July 2024); https://doi.org/10.1117/12.3019063

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available