Welcome to the Darshan project

This is the home page for  Darshan, a scalable HPC I/O characterization tool. Darshan is designed to capture an accurate picture of application I/O behavior, including properties such as patterns of access within files, with minimum overhead.  The name is taken from a Sanskrit word for “sight” or “vision”.

Darshan can be used to investigate and tune the I/O behavior of complex HPC applications.  In addition, Darshan’s lightweight design makes it suitable for full time deployment for workload characterization of large systems.  We hope that such studies will help the storage research community to better serve the needs of scientific computing.

Darshan was originally developed on the IBM Blue Gene series of computers deployed at the Argonne Leadership Computing Facility, but it is portable across a wide variety of platforms include the Cray XE6, Cray XC30, and Linux clusters.  Darshan routinely instruments jobs using up to 786,432 compute cores on the Mira system at ALCF.

You will find current news about the Darshan project posted below.   Additional documentation and details about the Darshan are available from the links at the top of this page.

Darshan 3.1.7 release is now available

Darshan version 3.1.7 is now available for release HERE! This version addresses a few bug fixes in the prior Darshan release and also contains a couple of new features:

  • Bug fix in handling of DXT module data in the darshan-convert utility
    • Reported by Mahzad Khoshlessan
  • Bug fix in darshan-parser backwards compatibility: Darshan logs generated by Darshan versions prior to 3.1.0 may have included STDIO counters that were not properly up-converted
    • Reported by Teng Wang
  • Bug fix to MiB reported in I/O performance estimate of darshan-job-summary when both POSIX and STDIO data present
    • Reported/fixed by Glenn Lockwood
  • Added Darshan wrapper for ‘__open_2()’ call, needed for properly instrumenting open operations with some versions of gcc/glibc
    • Reported by Cormac Garvey
  • Added an instrumentation module for the MDHIM key/val storage system
  • Added support for properly handling ‘rename()’, ‘dup()’, ‘fileno()’, and ‘fdopen()’ operations in Darshan

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan wins 2018 R&D 100 award

We are proud to announce that the Darshan team at Argonne National Laboratory has won a 2018 R&D 100 award!  This prestigious award is given to the 100 top technologies of the year as chosen by R&D Magazine.  We would like to sincerely thank the entire Darshan user community for supporting us and helping to make the project so successful!

For more information about the award please see the Darshan R&D 100 award news article at MCS.

Argonne National Laboratory’s R&D 100 award winning Darshan team: Kevin Harms (ALCF), Shane Snyder (MCS), Phil Carns (MCS), Rob Ross (MCS), and Rob Latham (MCS)
Phil Carns accepting the R&D 100 award at the 2018 R&D 100 banquet

FTP/download problems (resolved)

Update: the normal Darshan download links are now working again.

Our FTP server is down as of 03/29/2018 and we are not yet sure when it will be returned to service.  If you need a tar ball with the latest Darshan release, but our normal download links aren’t working, then you can use this link instead:

https://xgitlab.cels.anl.gov/darshan/darshan/repository/darshan-3.1.6/archive.tar.gz

That will generate a tar ball directly from the git repository tag.

Darshan 3.1.6 release is now available

Darshan version 3.1.6 is available for download here. This release addresses two bugs discovered in version 3.1.5:

  • Bug fix in mapping of Darshan’s ‘MPI_File_read_all’ wrapper to the underlying MPI symbol. This results in a crash if a dynamically-linked application preloads the Darshan library and calls ‘MPI_File_read_all’.
  • Bug fix for linker failures caused when an application links in a library that calls MPI externally (e.g., PETSc).
    • Reported by Bilel Hadri

We highly recommend upgrading from version 3.1.5 to 3.1.6 particularly for deployments on production systems, as these bugs could be disruptive for end users.

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan 3.1.5 release now available

Darshan version 3.1.5 is now available for download here.

The list of new features and bug fixes include:

  • Darshan’s MPI instrumentation method now intercepts both MPI and PMPI symbols to workaround MPI library bindings that are calling PMPI routines directly
    • Contributed in part by Chris Zimmer
  • darshan-util now includes an analysis tool for visualizing DXT traces: dxt_analyzer
    • ​Contributed by Alex Sim
  • Darshan now supports the DARSHAN_EXCLUDE_DIRS environment variable, which specifies a comma-separated list of paths to avoid instrumenting at runtime
    • Contributed by Cristian Simarro
  • Darshan’s regression test system now includes test harnesses for Cray systems @ the ALCF (Theta) and NERSC (Cori/Edison)
  • Bug fix in Darshan’s shutdown procedure leading to potentially negative timer values
  • Bug fix in Darshan’s autoconf scripts causing specified libbz2 locations to be ignored
    • Contributed by Glenn Lockwood
  • Bug fix in Darshan’s wrapping of H5get_libversion that was causing linker failures in some cases
    • Reported by Jialin Liu
  • Bug fix in darshan-merge utility related to logs containing DXT data
    • Reported by Glenn Lockwood

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan events at SC17

We encourage anyone interested in Darshan and related technology to attend the following events at SC17 in Denver:

Darshan pre-release 3.1.5-pre1 is now available

A new  pre-release version (3.1.5-pre1) of Darshan is available for download here. Please be aware that this pre-release is experimental and not recommended for full-time use in production yet. An official 3.1.5 release will be available in a few weeks.

The list of new features and bug fixes include:

  • Darshan’s MPI instrumentation method now intercepts both MPI and PMPI symbols to workaround MPI library bindings that are calling PMPI routines directly
    • Contributed in part by Chris Zimmer
  • darshan-util now includes an analysis tool for visualizing DXT traces: dxt_analyzer
    • Contributed by Alex Sim
  • Darshan now supports the DARSHAN_EXCLUDE_DIRS environment variable, which specifies a comma-separated list of paths to avoid instrumenting at runtime
    • Contributed by Cristian Simarro
  • Darshan’s regression test system now includes test harnesses for Cray systems @ the ALCF (Theta) and NERSC (Cori/Edison)
  • Bug fix in Darshan’s shutdown procedure leading to potentially negative timer values
  • Bug fix in Darshan’s autoconf scripts causing specified libbz2 locations to be ignored
    • Contributed by Glenn Lockwood

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.