carns

Quarterly Newsletter, October 2021

by

Project News:

New Microservices:

  • We are happy to introduce a new key/value microservice, called Yokan, to the Mochi framework. You can find more details in the Yokan documentation and Yokan GitHub repository. Yokan aims to provide state-of-the-art key/value storage capabilities on top of Margo, following the best practices of the Mochi methodology. It provides many backends, including BerkeleyDB, GDBM, LevelDB, LMDB, RocksDB, TKRZW, Unqlite, and a number of in-memory key/value stores. It was designed to be highly configurable and highly flexible, making it easy to configure databases using JSON, and to provide your own database implementation if the ones we offer don’t satisfy you. Yokan also provides C++ and Python APIs in addition to the usual C API.

Software updates:

  • Libfabric 1.13.2 has resolved multiple outstanding bugs that impacted Mochi, particularly with the RXM provider which is used on TCP and Verbs networks. Please try it out and report if you have any problems.
  • Mercury version 2.1.0rc2 is now available. This is very close to the final 2.1.0 release of Mercury and is the default version supported in the mochi-spack-packages repository. It includes a UCX network driver, improvements to the shared memory transport, new threading options, and miscellaneous bug fixes.
  • Margo version 0.9.6 has also been released; it includes support for the upcoming Mercury 2.1.0 and performance enhancements that take advantage of upcoming features in Argobots 1.2.

Platform support:

  • Please remember to refer to the Mochi platform configurations repository for suggested configurations for various platforms. We have recently updated several example Spack environment files. Feel free to contribute more!

Contribution policy:

  • The Mochi Contributor License Agreement (CLA) has been updated to streamline the process of contributing source code to the project. We have also installed GitHub action that will automatically prompt you to digitally agree to the CLA terms when you open your first pull request. Let us know if you have any questions.

New/Upcoming Publications:

  • Srinivasan Ramesh, Robert Ross, Matthieu Dorier, Allen Malony, Philip Carns and Kevin Huck. SYMBIOMON: A High Performance, Composable Monitoring Service. TO APPEAR in the 28th IEEE International Conference on High Performance Computing, Data, & Analytics (HiPC 2021)

Mochi selected as a 2021 R&D 100 finalist

by

UPDATE: Mochi was announced as a R&D 100 winner in the Software/Services category on October 21, 2021! We will post more details soon.

The Mochi project, a collaboration between Argonne National Laboratory, Carnegie Mellon University, Los Alamos National Laboratory, and The HDF Group, has been selected as a finalist for the 2021 R&D 100 Awards. The the R&D 100 Awards have served as the most prestigious innovation awards program for the past 58 years; their mission is to identify and honor the top 100 new technologies of the year. Winners will be announced in November 2021.

Quarterly Newsletter, July 2021

by

Publication news:

  • Pierre Matri and Robert Ross. “Neon: Low-Latency Streaming Pipelines for HPC”, to appear in IEEE Cloud 2021, Sept 5-10 2021.
    • Introduces a new Mochi service for stream processing
  • Stay tuned for more Mochi-related publications at SC21 in November. More details will be posted once the SC21 technical program is announced.

Recent development updates:

  • A proof-of-concept of UCX support in Mercury is available in the master-ucx version of Mercury in the Mochi Spack repository
    • Please contact us if you are interested in this capability; it is under active development and should be considered experimental at this time.
  • The git origin/main branch of Margo includes new safety checks to ensure compatible Argobots runtime parameters if Argobots is initialized outside of Margo. This will be available in an upcoming release after coordinating updates to other Mochi packages.
  • Both Mochi and Margo have new Contributor License Agreement (CLA) documents available online as of July 2021 with more relaxed language than the previous version. We will soon streamline these even further with online electronic forms that will be activated within the GitHub contribution process.

Debugging tips:

  • We have encountered several bug reports on Libfabric 1.13.0 in the last few days, especially with the RXM provider. Debugging is in progress, but in the mean time you may want to consider reverting to an earlier release if you encounter communication problems.
  • Recent libfabric releases also include a new PSM3 provider. PSM3 is not directly supported by Mercury / Mochi, but enabling it in libfabric may interfere with the performance of the traditional PSM2 provider. The libfabric package in the Mochi spack repository disables PSM3 by default for now to avoid this problem.

Quarterly newsletter, April 2021

by

New presentation materials:

GitHub migration complete:

New software releases:

  • Argobots 1.1
    • Underlying user-level threading package for Mochi
    • includes performance improvements, broader platform support, and new profiling and debugging capabilities (more on that later)
  • Mercury 2.0.1rc3
    • Underlying RPC communication package for Mochi
    • improved logging and several performance optimizations
    • final 2.0.1 release coming soon
  • Mochi-sdskv 0.1.12
    • Key/Value store microservice
    • Bedrock support
    • various packaging (cmake, pkgconfig, and dependency) improvements
  • Bedrock 0.2.1
    • Flexible service composition tool
    • various packaging (cmake, pkgconfig) improvements
  • Sonata 0.6.2
    • Document store microservice
    • various packaging (cmake) improvements

Performance regressions from previous quarterly newsletter resolved:

  • Power9 CPU mutex locking performance regression is resolved in Argobots 1.1
  • OmniPath network performance regression is resolved in Mercury 2.0.1rc3

New debugging/profiling/maintenance features:

  • Margo is now using munit for unit testing
    • Available in origin/main (or mochi-margo@main in Spack)
    • Coverage is limited for now but will be expanded over time
    • We will also be leveraging this frame work in additional components over time
  • Recent Argobots updates include multiple (optional) stack guard methods
    • See Argobots documentation or Spack package variants. Notable optoins:
      • “mprotect”: real time detection of stack overruns (with some performance overhead; just use this for debugging)
      • “canary”: lightweight deferred stack overrun detection (lighter weight, but will not report that a stack overflow occurred until shutdown)
  • margo_state_dump() function
    • Available in origin/main (or mochi-margo@main in Spack)
    • function that can be called at any time to dump point-in-time state to a text file or stdout for debugging purposes
    • includes Margo json configuration, Argobots configuration, current Argobots ES layout, Argobots performance profile, in flight RPC counts, stack dump for blocked user-level threads, etc. See https://github.com/mochi-hpc/mochi-margo/blob/main/doc/debugging.md for details.

The Mochi Github migration is complete

by

All Mochi source code repositories have been migrated to github.com at https://github.com/mochi-hpc/ as of March 22, 2021.

If you are already using spack to install Mochi components, please update your Mochi repository at your earliest convenience:

spack repo rm mochi
git clone https://github.com/mochi-hpc/mochi-spack-packages.git
spack repo add mochi-spack-packages

The package names have not changed; this will just enable you to retrieve new versions as they are released by updating your cloned copy of the mochi-spack-packages repo.

Mochi BoF at the ECP Annual Meeting

by

The Mochi team will be hosting a BoF entitled “Using Mochi to build data services: Overview and Updates” at the (virtual) ECP Annual Meeting on Tue. Apr 13, 2021 at 2:30 PM.

If you are an ECP project member attending the meeting, you can find more information about the BoF in the meeting agenda.

We will be providing an overview of Mochi, highlighting new capabilities, and offering sign-ups for one-on-one sessions for anyone who would like more detailed information or help.

Quarterly newsletter, January 2021

by

Platform notes:

Logistics:

  • All of the ANL-hosted git repositories (https://xgitlab.cels.anl.gov/sds/) will be moving within the next few months. We will communicate when that happens.
    • There are no changes to policy or access (in fact, access will likely change for the better); it’s just that the xgitlab.cels instance that we are using is being decommissioned
  • We are working on landing a Spack PR (https://github.com/spack/spack/pull/20273) that will introduce a “mochi-margo” package, maintained by us, to replace the out-of-date “margo” package
    • Once this is done, we will likely start upstreaming more packages that depend on margo-mochi

Mochi service development news:

  • Work continues on a new component called “Bedrock” that can be used to more easily bootstrap microservice compositions (https://xgitlab.cels.anl.gov/sds/bedrock).
    • Bedrock is already available, and we are in the process of updating existing services to use it.
    • You can think of bedrock as a general-purpose Mochi daemon that takes a JSON configuration file describing how to spin up embedded microservices
  • We are actively working on performance tuning of “Benvolio”, which you can think of as a runtime I/O delegation service (i.e. that provides a more generic version of MPI-IO aggregation capabilities). https://xgitlab.cels.anl.gov/sds/benvolio

Upcoming training events:

  • We plan to host a BoF session at this year’s (virtual) ECP annual meeting
    in mid-April, with a mechanism for people to sign up for one on one
    sessions for more detailed interaction. (https://ecpannualmeeting.com/)
  • Please let us know what other kinds of outreach/training you are interested in this year.