Prototype Outcomes

Prototypes

Data Engineering and Services

Component Documentation Code Description
Lambda-based Kerchunk Ingest Documentation Code Executing kerchunk workflows using AWS Lambda and SNS/SQS
XPublish Web Mapping Service (WMS) Documentation Code Retrieve web map tiles directly from environmental data
XPublish Enhancements (2024 IOOS Code Sprint) Documentation Code Plugins and backend enhancements to XPublish
RESTful Grids (2022 IOOS Code Sprint) Documentation Code Initial project investigating XPublish
Argo Workflows: Test QARTOD and IOOS Compliance Checker Documentation Code Use of standard IOOS tools within a cloud workflow environment
Argo Workflows: Kerchunk Ingest for NOS model data Documentation Code Executing kerchunk workflows within Argo Workflows platform
HRRR Aggregator Documentation Code Initial prototype to build large time-based aggregations by David Stuebe, Camus Energy
GRIB Index Aggregation Documentation Code Updated prototype to optimize large time-based aggregations for GRIB files by David Stuebe, Camus Energy

Infrastructure

Component Documentation Code Description
Kubernetes Documentation Configuration Demonstrated and learned from running K8s in AWS as a potential computing platform
Pulumi Infrastructure-as-Code Documentation Configuration Using Pulumi to initialize cloud resources needed to run kerchunk ingest
RabbitMQ Messaging Documentation Code Demonstrates platform-independent messaging
JupyterHub w/ Dask on K8s Documentation Configuration Initial experiments running Jupyterhub on K8s (later adopted Nebari)
Dask Gateway Documentation Configuration Running a Dask cluster on K8s

Demos and Notebooks