Welcome to the CODES project

The goal of the CODES project is use highly parallel simulation to explore the design of exascale storage architectures and distributed data-intensive science facilities.

Increasingly, science endeavors rely heavily on data management, analysis, and storage as part of the discovery process. To serve large communities of scientists, complex systems and instruments are deployed across multiple institutions to manage and analyze data produced from experiments, observational platforms, and computational simulation. Evaluating designs and coordinating deployment and operation of such a virtual data facility poses a significant challenge. An ability to simulate these environments would transform the approach taken to design, procurement, tuning, and upgrade of these facilities.

Our simulations build upon the Rensselaer Optimistic Simulation System (ROSS), a discrete event simulation framework that allows simulations to be run in parallel, decreasing the simulation run time of massive simulations to hours. We are using ROSS to explore topics including large-scale storage systems, I/O workloads, HPC network fabrics, distributed science systems, and data-intensive computation environments.

The CODES project is a collaboration between the Mathematics and Computer Science department at Argonne National Laboratory and Rensselaer Polytechnic Institute. We also collaborate with Lawrence Livermore National Laboratory for modeling HPC interconnect systems.

Posted in Uncategorized | Comments Off on Welcome to the CODES project

CODES @ SC16

We had a number of CODES related activities at Supercomputing this year! The research work presented used the CODES/ROSS framework to evaluate HPC network and storage systems, study application interference, enable efficient collective communication, and perform visual analysis of simulations. Details can be found at CODES@SC16.

Posted in Uncategorized | Comments Off on CODES @ SC16

Summer of CODES 2016!

Thanks everyone for attending this year’s Summer of CODES workshop! We had a great attendance and excellent talks. Slides will be posted in the coming days as I receive them, at the workshop website, http://press3.mcs.anl.gov/summerofcodes2016/.

soc-2016-group

Posted in Uncategorized | Comments Off on Summer of CODES 2016!

CODES 0.5.2 Released!

I am happy to announce the release of CODES 0.5.2! This release, prepared during this year’s Summer of CODES workshop (post incoming), is primarily a bug-fix release that includes a few minor API additions and updates to the latest ROSS API. Full release notes can be found at doc/RELEASE_NOTES in the release tarball.

Downloads can be found at http://www.mcs.anl.gov/projects/codes/downloads/.

Posted in Uncategorized | Comments Off on CODES 0.5.2 Released!

CODES 0.5.1 Released!

We are happy to announce the release of CODES 0.5.1! This release is a hotfix for latency calculation on the dragonfly, along with a few trivia missed in the past release. Downloads can be found at http://www.mcs.anl.gov/projects/codes/downloads/.

Posted in Uncategorized | Comments Off on CODES 0.5.1 Released!

CODES 0.5.0 Released!

We are happy to announce the release of CODES 0.5.0! It’s been a full year since the last content release and there have been a massive set of changes! See doc/RELEASE_NOTES for a full list – this post covers a few of the major ones. Downloads can be found at http://www.mcs.anl.gov/projects/codes/downloads/

  • codes-base and codes-net have been merged, greatly simplifying the build process. The new repository is at https://xgitlab.cels.anl.gov/codes/codes.
  • we’ve added a Slim Fly network topology simulator, corresponding to the Wolfe et al. paper “Modeling a Million-node Slim Fly Network using Parallel Discrete-event Simulation” at SIGSIM-PADS’16.
  • the dragonfly and torus networks have seen many improvements, including updates to the credit-based flow control mechanism, in-depth data gathering on both terminals and routers, and the ability to periodically sample the terminal/router states for gathering time-series data.
  • workload processing (and the MPI simulation layer) have been greatly enhanced, improving task to LP mapping flexibility and allowing concurrently running workloads.
  • a number of new APIs, including a “mapping context” construct for building more flexible implicit LP->LP mappings (now used in model-net, the local storage LP, and the resource LP) and a mechanism for RPC/callback-oriented event control flow (see the resource LP API and implementation for how this works).

Feedback is welcome!

Posted in Uncategorized | Comments Off on CODES 0.5.0 Released!

CODES 0.4.1 Released!

CODES 0.4.1 has been released! This is a hotfix release that fixes an incompatibility with recent ROSS revisions. The new version can be found at http://www.mcs.anl.gov/research/projects/codes/downloads/ .

Posted in Uncategorized | Comments Off on CODES 0.4.1 Released!

Summer of CODES 2015 slides available

The slides for (most of) the Summer of CODES 2015 talks have been made available. Check them out over at http://press3.mcs.anl.gov/summerofcodes2015/workshop-proceedings/.

Posted in Uncategorized | Comments Off on Summer of CODES 2015 slides available

Summer of CODES 2015

Thanks to all that attended the Summer of CODES 2015 Workshop! We had a strong first showing and a full schedule of engaging talks. We will post slides onto the workshop website (http://press3.mcs.anl.gov/summerofcodes2015/) in the coming days.

group

Posted in Uncategorized | Comments Off on Summer of CODES 2015

CODES 0.4.0 released!

CODES 0.4.0 has been released! You can find the release here.

Notable in this release include a large cleanup/refactoring of the codebase, improvements to error handling, numerous bugfixes, and improved documentation. Additionally, the workload generator API for IO and networking has been merged, though the overall implementation is still in experimental phase.

Full changelog (can also be found at doc/RELEASE_NOTES in the codes-base/codes-net tarballs):

codes-base

general:
==========
significant source reorganization / refactoring
refactor some private headers out of the public eye
dead code removal

documentation:
==========
improved example_heterogeneous example program
added configuration to example_heterogeneous showing two torus networks
reorganized files to prevent name collisions on OSX. Top-level docs other than
copyright now in doc directory
additions to best practice document

configurator:
==========
more stable file format for configurator output
ignore unrelated parameters passed into filter_configs
handle empty cfields in configurator

workloads:
==========
combined network and IO workload APIs into a single one
adding dumpi workload support in codes-workload-dump utility
workload dump utility option cleanup
renamed “bgp” workload generator to “iolang”, significant cleanups
put network workload ops in workload dump util
removing one of the dumpi libraries from the build. It was generating some unwanted dumpi files.
network workload API more fleshed out

utilities:
==========
configuration bug fixes for larger LP type counts
resource LP annotation mapping hooks
local storage model API switch to use annotations
better configuration error handling
hedge against precision loss in codes_local_latency (see codes.h)
use a different RNG than default for codes_local_latency
– prevents addition/removal of codes_local_latency calls from poisoning RNG
stream of calling model
added simple GVT-aware stack with garbage collection (see rc-stack.h)

codes-net

general:
==========
cleanup of much of the code base
more informative error for failure to find modelnet lps
removed redundant include directory on install (was ‘install/codes/codes/*.h’)

documentation:
==========
reorganized files to prevent name collisions on OSX. Top-level docs other than
copyright now in doc directory
updated code documentation
fix linker error in certain cases with codes-base
tweaked config error handling

networks:
==========
fix to loggp latency calculation when using “receive queue”
made torus lps agnostic to groups and aware of annotations
miscellaneous fixes to dragonfly model
updates to simplep2p: support for having different latency/bw at sender &
receiver end. See src/models/networks/model-net/doc/README.simplep2p.txt
minor fixes to usage of quickhash in replay tool
fixed RNG reverse computation bug in loggp
fixed swapped arguments in round-robin scheduler causing short circuit

workloads:
==========
minor changes to dumpi trace config files
resolving minor bug with reverse computation in dumpi traces
Updating network trace code to use the combined workload API
Adding synthetic traffic patterns (currently with dragonfly model)
Adding network workload test program for debugging
Updating MPI wait/wait_all code in replay tool

Posted in Uncategorized | Comments Off on CODES 0.4.0 released!

CODES 0.3.0 released!

CODES 0.3.0 is released! This marks the first formal release of the codes-base and codes-net projects. You can find the release on the download page.

Against recent repository revisions, this release includes more complete and cleaned up documentation.

Posted in Uncategorized | Comments Off on CODES 0.3.0 released!