Welcome to the Darshan project

This is the home page for  Darshan, a scalable HPC I/O characterization tool. Darshan is designed to capture an accurate picture of application I/O behavior, including properties such as patterns of access within files, with minimum overhead.  The name is taken from a Sanskrit word for “sight” or “vision”.

Darshan can be used to investigate and tune the I/O behavior of complex HPC applications.  In addition, Darshan’s lightweight design makes it suitable for full time deployment for workload characterization of large systems.  We hope that such studies will help the storage research community to better serve the needs of scientific computing.

Darshan was originally developed on the IBM Blue Gene series of computers deployed at the Argonne Leadership Computing Facility, but it is portable across a wide variety of platforms include the Cray XE6, Cray XC30, and Linux clusters.  Darshan routinely instruments jobs using up to 786,432 compute cores on the Mira system at ALCF.

You will find current news about the Darshan project posted below.   Additional documentation and details about the Darshan are available from the links at the top of this page.

New experimental version of Darshan available for instrumenting non-MPI applications

An experimental pre-release of Darshan is now available that enables instrumentation of non-MPI workloads. It can be downloaded here. It is NOT recommended to use this version in production until we have had more time for users to test it.

See the darshan-runtime documentation (located in darshan-runtime/docs from the top-level Darshan repo) for more information on how to build Darshan without MPI support and also how to enable non-MPI instrumentation at application runtime.

Note that this instrumentation method only works on dynamically-linked executables — Darshan still does not support instrumentation of statically-linked non-MPI executables.

We encourage users that are interested in characterizing I/O in non-MPI contexts to try out this new functionality and let us know about any issues or questions you might have! Depending on user experience, we will try to get a release of this software suitable for production deployment soon.

Darshan at SC19 recap

In case you missed any of it, here’s a list of Darshan-related activities from SC that maybe of interest to the community:

Darshan version 3.1.8 now available

Darshan 3.1.8 is now available for download here.

This release introduces a new trace triggering mechanism that allows users to specify triggers that dictate which files are traced using Darshan’s tracing module, DXT. Users just need to provide Darshan a configuration file describing the triggers and Darshan will decide at runtime which files to store trace data for. Types of triggers include file- and rank-based triggers (based on regex patterns), as well as file access characteristics triggers (to trace based on frequency of small or unaligned I/O accesses). Please refer to darshan-runtime documentation on the DXT module for more details.

Note that full tracing is disabled by default in Darshan and this release does not change that — this is just a mechanism to allow DXT users more control over tracing.

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan 3.1.7 release is now available

Darshan version 3.1.7 is now available for release HERE! This version addresses a few bug fixes in the prior Darshan release and also contains a couple of new features:

  • Bug fix in handling of DXT module data in the darshan-convert utility
    • Reported by Mahzad Khoshlessan
  • Bug fix in darshan-parser backwards compatibility: Darshan logs generated by Darshan versions prior to 3.1.0 may have included STDIO counters that were not properly up-converted
    • Reported by Teng Wang
  • Bug fix to MiB reported in I/O performance estimate of darshan-job-summary when both POSIX and STDIO data present
    • Reported/fixed by Glenn Lockwood
  • Added Darshan wrapper for ‘__open_2()’ call, needed for properly instrumenting open operations with some versions of gcc/glibc
    • Reported by Cormac Garvey
  • Added an instrumentation module for the MDHIM key/val storage system
  • Added support for properly handling ‘rename()’, ‘dup()’, ‘fileno()’, and ‘fdopen()’ operations in Darshan

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan wins 2018 R&D 100 award

We are proud to announce that the Darshan team at Argonne National Laboratory has won a 2018 R&D 100 award!  This prestigious award is given to the 100 top technologies of the year as chosen by R&D Magazine.  We would like to sincerely thank the entire Darshan user community for supporting us and helping to make the project so successful!

For more information about the award please see the Darshan R&D 100 award news article at MCS.

Argonne National Laboratory’s R&D 100 award winning Darshan team: Kevin Harms (ALCF), Shane Snyder (MCS), Phil Carns (MCS), Rob Ross (MCS), and Rob Latham (MCS)

Phil Carns accepting the R&D 100 award at the 2018 R&D 100 banquet

FTP/download problems (resolved)

Update: the normal Darshan download links are now working again.

Our FTP server is down as of 03/29/2018 and we are not yet sure when it will be returned to service.  If you need a tar ball with the latest Darshan release, but our normal download links aren’t working, then you can use this link instead:

https://xgitlab.cels.anl.gov/darshan/darshan/repository/darshan-3.1.6/archive.tar.gz

That will generate a tar ball directly from the git repository tag.

Darshan 3.1.6 release is now available

Darshan version 3.1.6 is available for download here. This release addresses two bugs discovered in version 3.1.5:

  • Bug fix in mapping of Darshan’s ‘MPI_File_read_all’ wrapper to the underlying MPI symbol. This results in a crash if a dynamically-linked application preloads the Darshan library and calls ‘MPI_File_read_all’.
  • Bug fix for linker failures caused when an application links in a library that calls MPI externally (e.g., PETSc).
    • Reported by Bilel Hadri

We highly recommend upgrading from version 3.1.5 to 3.1.6 particularly for deployments on production systems, as these bugs could be disruptive for end users.

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.