The Vera C. Rubin Observatory, currently under construction in Chile, will conduct a vast astronomical survey of our dynamic Universe starting in 2022.
They plan to collect 500 petabytes of image data by observing the skies continuously for 10 years and produce nearly instant alerts for objects that change in position or brightness every night.
In addition to astronomical data, their dataset will include DevOps, IoT, and real-time monitoring data.
DBTA recently held a webinar with Dr. Angelo Fausti, software engineer, Vera C. Rubin Observatory, who discussed how a time series database has the versatility to address their needs.
Because relational databases are not optimized for time series data, Fausti explained, the observatory chose InfluxDB time series relational database to timestamp data and analyze it.
The platform can ping scientists when there is an update via Slack, according to Fausti, which keeps the observatory abreast of any developing situations.
InfluxDB is a complete solution, he said. It offers:
- A time series database, has an SQL-like query language, and provides an HTTP API
- Chronograf UI for time series visualization written in Go and React.js
- Kapacitor framework written in Go for processing, monitoring, and alerting on time series data
- Generic and flexible
- OSS we did some customizations to Chronograf
- Chronograf dark mode (important if you work at the observatory at night)
“Astronomers working on telescopes have to monitor different things depending on what their assignments are. Chronograf gives us the flexibility to create dashboards that suit our needs on different situations,” said Tiago Ribeiro, scheduler scientist at the Vera C. Rubin Observatory.
The scientists were able to have sensors publishing information in many streams, and from multiple physical locations, according to Fausti. Kapacitor has been critical in sounding the alarm when the system is not behaving correctly.
The next steps for the observatory include:
- Migration to InfluxDB 2.0
- Waiting for annotations to be implemented in 2.0
- Testing the solution with increasing data volume and variety as more subsystems of the Telescope are put together
- Data replication to our data facility/cloud
- Which resources are needed to store the raw data for the lifetime of the project?
- Other problems to solve: observatory electronic log system, astronomical time-series, etc.
An archived on-demand replay of this webinar is available here.