Next-gen tools for big scientific data: ARM data center example

Ranjeet Devarakonda; Kyle Dumas; Sheman Beus; Everett Rush; Bhargavi Krishna; Rob Records; Giri Prakash

doi:10.1109/BigData.2016.7841078

2016 IEEE International Conference on Big Data (Big Data)

Next-gen tools for big scientific data: ARM data center example

Year: 2016, Pages: 3968-3970

DOI Bookmark: 10.1109/BigData.2016.7841078

Authors

Ranjeet Devarakonda, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA
Kyle Dumas, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA
Sheman Beus, ARM Climate Research Facility, Pacific Northwest National Laboratory, PO Box 999, Richland, WA 99352
Everett Rush, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA
Bhargavi Krishna, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA
Rob Records, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA
Giri Prakash, ARM Climate Research Facility, Oak Ridge National Laboratory, 1 Bethel Valley Rd, Oak Ridge, TN USA

Abstract

The Atmospheric Radiation Measurement (ARM) Climate Research Facility (www.arm.gov) provides atmospheric observations from diverse climatic regimes around the world. Currently, ARM archives over 22 million user assessable data files, primarily stored in NetCDF file format, with total data volumes close to one Petabyte. In this paper, we will discuss how ARM is currently storing, distributing, cataloging and visualizing such large volumes of multi-dimensional climate observations and model data and also describe their future plan.

Like what you’re reading?

Already a member?

Get this article FREE with a new membership!

HPC infrastructure to support the next-generation ARM facility data operations
2016 IEEE International Conference on Big Data (Big Data)
Exploring the evolution of pressure-perturbations to understand atmospheric phenomena
2017 IEEE Pacific Visualization Symposium (PacificVis)
Detecting Outliers in Streaming Time Series Data from ARM Distributed Sensors
2018 IEEE International Conference on Data Mining Workshops (ICDMW)
A Collaborative Effort to Improve Lossy Compression Methods for Climate Data
2019 IEEE/ACM 5th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD-5)
Big Federal Data Centers Implementing FAIR Data Principles: ARM Data Center Example
2019 IEEE International Conference on Big Data (Big Data)
Provenance–aware workflow for data quality management and improvement for large continuous scientific data streams
2019 IEEE International Conference on Big Data (Big Data)
On Preserving Scientific Integrity for Climate Model Data in the HPC Era
Computing in Science & Engineering
Enhancing Discoverability and Management of Atmospheric Data at Scale: Solutions from the ARM Data Center
2024 IEEE International Conference on Big Data (BigData)
Data Workbench For Earth and Atmospheric Science Research Community
2024 IEEE International Conference on Big Data (BigData)
The Gather Join: Efficiently Aligning Atmospheric Datasets in Apache Spark
2024 IEEE International Conference on Big Data (BigData)

Next-gen tools for big scientific data: ARM data center example

Authors

Abstract

Related Articles