Welcome to the Mochi project

The modern DOE scientific computing portfolio consists of a rich ecosystem of simulation, data analytics, and learning applications, with many distinct data management and analysis needs. The objective of the Mochi project is to design methodologies and tools that allow for the rapid development of distributed data services in support of DOE science. An important aspect of Mochi is composition: common capabilities such as communication, data storage, concurrency management, and group membership are provided under Mochi along with building blocks such as BLOB and key-value stores. These building blocks are mixed together to provide specialized service implementations catering to specific platforms and science needs. Current Mochi research directions include unifying management of disparate data classes from scientific campaigns and applying learning and artificial intelligence to improve the adaptability of data services on heterogeneous DOE platforms.
The Mochi project is a collaboration between Argonne National Laboratory, Los Alamos National Laboratory, Carnegie Mellon University, and the HDF Group. However, Mochi is also bigger than just these partners: Mochi is an open ecosystem enabling the development of a variety of services both within the DOE and internationally.

Quarterly Newsletter, January 2024

The Mochi quarterly meeting for Thursday, January 25, 2024 has been cancelled. Please reach out to us on the mailing list or Slack space if you have anything that you would like to discuss with the Mochi team. Otherwise we hope to see you at our next quarterly meeting on April 25.

Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.

Mochi updates

  • Software updates
    • Mofka version 0.0.2 was released on January 19, 2024. Mofka is a distributed event streaming service with semantics and data models similar to that of Kafka, but geared towards scientific computing on HPC platforms. It is still in an early alpha stage but we invite questions, feedback, and comments. We will cover Mofka in more detail in a future meeting.
    • Mercury version 2.3.1 was released on October 2.3.1. Mercury is the RPC framework used by Mochi for all communication and data transfer. This point release includes a collection of bug fixes and optimizations as well as support for more Slingshot VNI configurations.
  • Platform updates
    • The Mochi team is now using the ALCF Gitlab CI infrastructure to execute nightly performance regression tests on the Polaris system at the ALCF. We hope to expand this testing in the future. For now its primary objective is to monitor Slingshot network performance across not only Mochi updates but also HPE system software updates.

Quarterly meeting and newsletter, October 2023

Please join us for the next Mochi quarterly meeting on Thursday, October 26, 2023, at 10am CT. Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.


Microsoft Teams meeting
Join on your computer or mobile app
Click here to join the meeting
Or call in (audio only)
+1 630-556-7958,,254649841#

Mochi updates and agenda items

  • Upcoming Publications and Presentations
    • “Mochi: A Case Study in Translational Computer Science for High-Performance Computing Data Management” (under preparation for an upcoming issue of IEEE Computing in Science and Engineering)
    • If you are attending IEEE Cluster 2023 in Santa Fe, please consider stopping by the REX-IO workshop on Tuesday October 31 for the keynote presentation “Anticipating and Adapting to Change in HPC Storage” by Phil Carns.
  • New Mochi Microservices
    • Matthieu Dorier will present an overview of Warabi, a new blob storage microservice. Warabi has similarities to Bake, but has been designed from the ground up with a cleaner, more comprehensive API and seamless integration with the Bedrock ecosystem.
  • HPE Slingshot Status Update
    • Communicating on a Slingshot network requires access to a Virtual Network Interface, or VNI, to authorize communication across processes. You may need to take additional steps to configure the VNI depending on your use case.
      • Communicating across processes that were launched together (e.g. in the same srun or mpiexec invocation):
        • Mercury and thus Mochi will use the same VNI allocated for use by MPI with no additional configuration needed.
        • You may need to use a “–single-node-vni” argument to mpiexec or a “–network=single_node_vni” argument to srun, depending on your platform, to make sure that a VNI is allocated even if the launcher believes that all processes will be executing on the same node.
      • Communicating across independently-launched processes within a job:
        • On the Aurora or Sunspot systems at ANL, no additional configuration is needed.
        • On HPE/SLURM based systems (i.e. Frontier and Perlmutter) additional configuration is needed, because these systems utilize a unique VNI for each job step by default. You can instruct Mercury to instead use a job-level VNI by passing a special value of “0:0” in the “auth_key” field of the Mercury json configuration in Mochi. This feature is already available in mercury@master but will also be available in the next point release. In addition, you must also enable the job-level VNI with the –network=job_vni option to the sbatch command or as a directive at the top of your job script.
      • Communicating across jobs:
        • We are still working with HPE on a general solution to enable communication across jobs.
  • General platform updates:

Quarterly meeting and newsletter, July 2023

Please join us for the next Mochi quarterly meeting on Thursday, July 27, 2023, at 10am CT. Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.


Microsoft Teams meeting
Join on your computer or mobile app
Click here to join the meeting
Or call in (audio only)
+1 630-556-7958,,254649841#

Mochi updates and agenda items

  • Recent presentations
    • The Mochi team presented two seminars in June 2023 as part of the Mathematics and Computer Science Division’s CS seminar series. These seminars provide an overview of the state of Mochi and how it can be used in 2023. The first covers high-level motivation, concepts, and key technologies, while the second describes the Mochi methodology for composable data services and highlights success stories in domain-specific data services (HEPnOS) and elastic in situ visualization (Colza).
      • Philip Carns. “Mochi Project Overview: the Democratization of Data Services in HPC”, CS seminar series, Argonne National Laboratory Mathematics and Computer Science division, June 13, 2023. ABSTRACT PDF VIDEO
      • Matthieu Dorier. “Mochi in Practice: Data Services for High-Energy Physics and Elastic In Situ Visualization Workflows”, CS seminar series, Argonne National Laboratory Mathematics and Computer Science division, June 20, 2023. ABSTRACT PDF VIDEO
  • Recent tutorials
    • Matthieu Dorier, Philip Carns, and Marc-André Vef presented the following tutorial on May 21 at ISC High Performance 2023:
  • New features
    • Yokan now enjoys 4 new families of functions: yk_fetch , yk_doc_fetch, yk_iter and yk_doc_iter (each with variants to access multiple key/value pairs or documents at once). These functions are equivalent to yk_get, yk_doc_load, yk_list_keyvals and yk_doc_list, respectively, but take a callback that is invoked on each key/value pair or document, instead of taking a buffer in which the key/value pair or document is copied. These functions allow for fewer memory copies and simpler code (no need for the caller to manage their own buffer or call other functions to query the size of values/documents first). The _iter functions also provide automatic pipelining and batching.
    • Mercury 2.3.0 is out now, including several notable performance enhancements for libfabric and CXI:
      • new “multi-recv” optimization to improve RPC throughput
      • avoid performance degradation in FI_SOURCE
      • use WAIT_FD for graceful idling on Slingshot (CXI) transports
  • HPE Slingshot status update
    • Mercury support for Slingshot (CXI) is feature complete and performing well with Mercury 2.3.0, but there are some important usability issues to be aware of regarding Virtual Network Interfaces (VNIs). VNIs are a mandatory method for Slingshot network access control between compute nodes.
    • The default job launcher on HPE systems will automatically provision a VNI for MPI. Mercury will inherit and use this same VNI without any additional action on your part.
    • However, this default, launcher-provided VNI is not sufficient for communicating across MPI jobs or among manually-launched processes.
    • We are in communication with HPE about this issue, but they are still working on a general solution. If you encounter problems, please alert your facility or vendor contacts and let us know about your experience!

Mochi CS seminars at Argonne National Laboratory

The Mochi team presented the following two seminars in June 2023 as part of the Mathematics and Computer Science Division’s CS seminar series:

  • Philip Carns. “Mochi Project Overview: the Democratization of Data Services in HPC”, CS seminar series, Argonne National Laboratory Mathematics and Computer Science division, June 13, 2023. ABSTRACT PDF VIDEO
  • Matthieu Dorier. “Mochi in Practice: Data Services for High-Energy Physics and Elastic In Situ Visualization Workflows”, CS seminar series, Argonne National Laboratory Mathematics and Computer Science division, June 20, 2023. ABSTRACT PDF VIDEO

The first is a high-level presentation of the motivation, concepts, and key technologies that make Mochi possible. The second describes the Mochi methodology for composable data services and highlights success stories in domain-specific data services (HEPnOS) and elastic in situ visualization (Colza).

Together, these presentations give a nice overview of the state of Mochi and how it can be used in 2023. See the links above for more detailed abstracts, slides, and a video recordings of the presentations.

Quarterly meeting and newsletter, April 2023

Please join us for the next Mochi quarterly meeting on Thursday, April 27, 2023, at 10am CT. Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.


Microsoft Teams meeting
Join on your computer or mobile app

Click here to join the meeting

Or call in (audio only)
 

Mochi updates and agenda items


Quarterly meeting and newsletter, January 2023

Please join us for the next Mochi quarterly meeting on Thursday, January 26, 2023, at 10am CT. Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.


Microsoft Teams meeting
Join on your computer or mobile app

Click here to join the meeting

Or call in (audio only)
+1 630-556-7958,,254649841#   United States, Big Rock

Phone Conference ID: 254 649 841#


We plan to discuss the following topics at this meeting:

  • Call for lightning presentations:
    • Do you have something that you would like to present at the Mochi quarterly meeting? We would love to hear about interesting services you have built using Mochi technology, performance results, challenges and obstacles, or all of the above! It may be short notice for this meeting, but please let us know if you would like to share a presentation this week or request a slot for a future meeting.
  • Elasticity support in Margo (Matthieu Dorier):
  • Call for feedback on Mochi tutorial topics
    • The Mochi team is planning to introduce new tutorial material this year (venues TBA).
    • Upcoming tutorials will focus on hands-on exercises that use containers and Mochi service templates to get up and running quickly.
    • What suggestions do you have for us on what points we should cover as we develop this new material?
    • Examples of previous Mochi tutorials can be found at https://www.mcs.anl.gov/research/projects/mochi/tutorials/

Quarterly meeting and newsletter, October 2022

Please join us for the next Mochi quarterly meeting on Thursday, October 27, 2022, at 10am CT. Mochi quarterly meetings are a great opportunity to learn about community activities, share best practices, get help with problems, and find out what’s new in Mochi.

Please suggest agenda items on the Mochi slack space or the [email protected] mailing list.


Microsoft Teams meeting
Join on your computer or mobile app
Click here to join the meeting
Or call in (audio only)
+1 630-556-7958,,254649841#   United States, Big Rock

Phone Conference ID: 254 649 841#


We plan to discuss the following topics at this meeting:

Quarterly Newsletter, July 2022

In lieu of our usual software update this quarter, we would instead like to invite the community to join us for the July 28 quarterly meeting to discuss ideas for how to improve the process of bootstrapping, testing, and validating Mochi software environments. This process is more ad-hoc than we would like for it to be right now; anything we can do to formalize and streamline would be a big help in the long run.

If you have any suggestions or comments to share (on this topic or anything else Mochi-related) please connect to the following meeting on Thursday, July 28, at 10:00am CT:

Mochi Quarterly meeting

Click here to join the meeting

Or call in (audio only)

+1 630-556-7958,,254649841#  United States, Big Rock

Phone Conference ID: 254 649 841#

Find a local number | Reset PIN

Learn More | Meeting options

Please join our mailing list (see link on the right side of this web site) if you would like to suggest agenda items in advance.

Building Custom Data Services with Mochi BoF, 2022

Team Mochi, along with special guests Philip Davis of the University of Utah and Chris Kelly of Brookhaven National Laboratory, hosted a BoF session entitled “Building Custom Data Services with Mochi” at the 2022 ECP Community BoF Days. Thank you everyone for participating! The slides are now available online on the Mochi Tutorials page.

UPDATE: a full video event is also now available from the tutorials link provided above.

Quarterly Newsletter, April 2022

New tools

  • Mochi-json-vis
    • https://github.com/mochi-hpc/mochi-json-vis
    • A command-line tool that can be used to generate a visual representation of a Mochi Bedrock configuration.
    • This can be helpful to sanity check or better understand service configuration details such as the mapping of providers to execution streams.

Software updates

  • Mochi-thallium 0.10.1 (C++ bindings to Mochi)
    • Adds support for timer_callback
    • Adds logger class and logging functionality
    • Adds access to margo’s underlying configuration, pools, and xstreams
  • Mochi-bedrock 0.4.1 (service configuration framework)
    • Ability to initialize the server with a JX9 script instead of a JSON configuration

Publications

  • Matthieu Dorier, Zhe Wang, Utkarsh Ayachit, Shane Snyder, Robert Ross, Manish Parashar. “Colza: Enabling Elastic In Situ Visualization for High-performance Computing Simulations.” in Proceedings of the 36th IEEE International Parallel & Distributed Processing Symposium (IPDPS 2022) (TO APPEAR)
  • Bradley Settlemyer, George Amvrosiadis, Philip Carns, and Robert Ross. “It’s time to talk about HPC storage: Perspectives on the past and future.” Computing in Science & Engineering, 23(6):63–68, 2021. https://ieeexplore.ieee.org/document/9658238

Upcoming events

Building Custom Data Services with Mochi (public BoF)

May 12th, 11:00 AM eastern time

We will provide general updates on the Mochi project, highlight key capabilities related to service composition and key/value stores, and share work from guest speakers about the Mochi messaging layer and successful Mochi use cases:

  • Mercury: platform updates and optimizations for RPC and RDMA communication (Jerome Soumagne, The HDF Group)
  • Chimbuko: scalable application performance analysis and provenance (Chris Kelly, Brookhaven National Laboratory)
  • DataSpaces: extreme-scale data management framework (Philip Davis, University of Utah)

To register, follow this link, expand the Mochi BoF description, and click “Register” — this should provide you with a Zoom link to attend the BoF: https://www.exascaleproject.org/event/ecp-community-bof-days-2022/