CloudMap
Brian Calder (UNH)
CloudMap I – Overview
Collection of data by autonomous systems will add significantly to the already burdensome data processing task. Efficiency in processing is therefore even more important than usual, and with modern data management trending towards cloud-based systems, it is imperative, from a cost and efficiency perspective, that data processing follows the data trends of autonomy. Hence, the development of cloud-native, scalable algorithms will be key to supporting these requirements.
CloudMap I – Accomplishments (2021-2022)
CloudMap aims, in this first year, to understand the performance costs for fundamental elements of the bathymetric processing task through deployment and analysis of an open source bathymetric processing package developed at NOAA; our ultimate goal is to scope the design of a cloud-native processing system that would be able to take advantage of the unique properties, particularly scalability, of the cloud environment.
In the current reporting period, Calder, Plumlee, and Cui have begun the investigation of desktop-system performance in the cloud through a series of experiments to benchmark a specific processing system on different cloud compute platforms, and with different storage options.
In addition to testing current systems, work was conducted on the preliminary design of a cloud-native processing system.
CloudMap II – Overview
Driven by pressures on berthing on ships of exploration, particularly with more vehicles on board to expand the exploration potential of a single platform, along with the desire for better visibility, discovery, and usability of data, there has been significantly increased interest in the last few years on the potential for tele-processing of data. That is, providing access to the data store on the ship from shore, avoiding the requirement to have data processors on ship; or, providing transport of the data to shore into a cloud processing environment so that multiple processors can access it and turn around products at the operational tempo of the assets at sea.
This project will first investigate how to build a “best in breed” conventional processing system for cloud deployment and allow for field testing of these ideas to examine the potential efficiency gains and concept of operations for shore-side processing of data. Second, we will investigate development of an experimental cloud-native bathymetry processing scheme which can scale as required in order to maintain a given processing rate, allowing the code to maintain a real-time “best estimate of depth” surface for the data being collected. At the same time, we will investigate how to maintain a consistent estimation state where data is added (e.g., as more data is collected) or removed (e.g., by editing) from multiple users in real time.
CloudMap II – Accomplishments (2022-2023)
At the start of the current reporting period, Calder and Worden continued the analysis of the performance timing for the “Desktop in the Cloud” experiment introduced in the initial year’s work, confirming and extending the results and conclusions from the preliminary analysis. Meanwhile, Calder and Plumlee continued with the design of the proposed cloud-native processing system. These two activities complete the work from year one of the effort. Calder then completed the work on definition of a “best in breed” system of this type, as outlined in the proposal for the current year, but further intensive investigation of scaling and deployment was not conducted since it would have been unproductive given the performance issues and economic costs determined during the prior work.
Subsequently, Calder, Miles, and Beregovyi have commenced the build-out of the microservice-structured cloud-native processing system. This has included development of the underlying support system to allow easy deployment of the system components. Calder has also developed a stateless version of the CHRT bathymetric estimation algorithm which can be used to (re-)compute depths in any given area. Finally, Beregovyi has begun work on the visualization component of the system, adapting a previous virtual reality point cloud editing system for streaming data, and higher performance. This provides approximately 75% of the required services (by module), although further development, testing, and integration is required.
CloudMap III – Overview
Despite decades of growth and innovation in cloud-based methods and distributed systems, software to process and manage bathymetric data remains stubbornly desktop-based and limited in processing performance. Prior investigation demonstrated that cloud-native algorithms are required to fully benefit from the resources and particularly the scalability of cloud-based systems. The CloudMap project seeks to construct such a cloud-native processing system, link it to an advanced VR-based user interface and data interaction system, and demonstrate performance and scaling with full-scale datasets.
