Archive

Archive for the ‘Uncategorized’ Category

“I/O characterization of large-scale applications with Darshan” presented at SDAV meeting

March 20th, 2014
Comments Off

A poster and presentation entitled “I/O characterization of large-scale applications with Darshan” was presented at the SciDAC Institute of Scalable Data Management, Analysis and Visualization All Hands Meeting in February 2014 at Georgia Tech.  The poster and presentation are available for download on the publications page.

Uncategorized

ALCF I/O Data Repository update

February 18th, 2014
Comments Off

The ALCF I/O Data Repository has been updated to include all of the Darshan logs collected on Intrepid from October through December of 2013.  This brings the total log count in the repository to 156,338.

Uncategorized

2012-2013 Darshan logs added to the the ALCF I/O Data Repository

October 29th, 2013
Comments Off

Anonymized versions of all Darshan logs collected on the Intrepid Blue Gene/P system at the Argonne Leadership Computing Facility from January 1, 2012 through October 22, 2013 are now available for download as part of the ALCF I/O Data Repository.

Darshan is a scalable HPC I/O characterization tool that collects concise I/O access pattern information from large-scale production applications.  The Darshan data provided in the ALCF I/O Data Repository includes:

  • I/O characterization from 152,167 unique production application runs*
  • over 721 million core hours of execution time
  • 31 PiB of I/O activity
  • examples of application runs with up to 163,840 processes
  • examples of application runs that accessed up to 204 TiB of data

More information about how to use the data can be found  at http://www.mcs.anl.gov/research/projects/darshan/data/.

* note: previously announced log file count of 195,233 was in error, but all other statistics are accurate

Uncategorized

Darshan-Ruby 1.1

September 10th, 2013
Comments Off

Matthieu Dorier of ENS Cachan Brittany and IRISA has released Darshan-Ruby 1.1, including support for Darshan 2.2.8 and a gem installation package.  Darshan-Ruby provides object-oriented Ruby bindings for analysis of Darshan log files.

Uncategorized

Darshan 2.2.8 release

September 3rd, 2013
Comments Off

Darshan 2.2.8 is now available for download on the Darshan web site. This release contains a number of bug fixes for problems recently reported on the mailing list. It also includes a few new tools for splitting up Darshan logs based on files opened by the application. The full change log is listed below:

darshan-2.2.8

  • WARNING: please note that the Darshan module file for Cray environments has been updated, especially in the DARSHAN_POST_LINK_OPTS variable. Please update your module file accordingly when upgrading from 2.2.7 or earlier on Cray platforms.
  • Improved ability to analyze I/O activity related to particular files opened by an application
    • darshan-summary-per-file.sh script to generate a separate pdf summary for each file opened by an application. Developed by Rob Latham.
    • Added –file-list and –file-list-detailed options to darshan-parser to list files opened by an application along with brief statistics
    • Added –file option to darshan-convert to filter out activity for a specific file from a Darshan log
  • Add wrappers for POSIX AIO operations (fixes tracking of underlying POSIX operations resulting from nonblocking MPI-IO operations. Bug reported by David Shrader.)
  • Fix compile-time errors when Darshan is configured for use with MPICH 3.x installations with const correct MPI function prototypes. Also affects MPICH 1.5+ installations with optional const support. Reported by Yushu Yao.
  • Fix segmentation fault when using LD_PRELOAD instrumentation on programs that use MPI_Init_thread() rather than MPI_Init(). Reported by Myriam Botalla.

Uncategorized

Darshan 2.2.7 release

June 20th, 2013
Comments Off

Darshan 2.2.7 is now available for download.

Changelog:

darshan-2.2.7

  • Updated Cray installation documentation for cleaner integration in Cray environments
  • Fix bug that recorded incorrect device ID (and therefore incorrect mount point mapping) if stat() was called before open() on a file
  • Store version number of the Darshan runtime library in the log file metadata (see lib_ver in darshan-parser output)

Cray users in particular should take note of the revised Darshan installation recipe, which is available both on the documentation section of the Darshan web site and in the darshan-runtime/doc/ directory in the source. The installation procedure for Cray systems has been simplified significantly from previous releases.

Uncategorized

Updated data repository documentation

June 18th, 2013
Comments Off

The documentation for the public ALCF I/O Data Repository has been updated.  Please note that there is a new technical report that contains an updated description of the data and a suggested acknowledgement to include with publications that use the data.

Uncategorized

Darshan 2.2.6 release

April 23rd, 2013
Comments Off

Darshan 2.2.6 has been released.  This release includes performance enhancements as well as various minor bug fixes and compatibility improvements.

Changelog:

darshan-2.2.6

  • Bug fixes:
    • make sure to honor user-specified hints passed in at runtime via the DARSHAN_LOGHINTS env variable.
    • include fread and fwrite in darshan-job-summary.pl read and write counts
    • fix segmentation fault on invalid arguments to darshan-parser
  • Performance enhancements:
    • collect mount point information at rank 0 and broadcast to all
    • processes to avoid excess file system traffic on startup
    • change default MPI-IO hints for writing log file to romio_no_indep_rw=true and cb_nodes=4 to improve log creation performance
  • Install libdarshan-util and headers during installation process for darshan-util; patch contributed by Matthieu Dorier
  • Detect PMPI support at link time when using compiler scripts produced by the darshan-gen-* utilities. This avoids link problems when Darshan compiler wrappers are used with the ADIOS dummy MPI library; reported by Jingqing Mu
  • Support MPICH_{CC/CXX/F77} environment variables in compiler scripts produced by the darshan-gen-* utilities
  • Add instrumentation of startup time, enabled via DARSHAN_INTERNAL_TIMING environment variable
  • Rename cp-shutdown-bench test utility to darshan-shutdown-bench and enable benchmarking hooks in library by default so that darshan-shutdown-bench can be used with any Darshan installation
  • Remove deprecated –enable-st-dev-workaround configure option
  • Misc. documentation updates

Uncategorized

Darshan-Ruby

April 15th, 2013
Comments Off
Matthieu Dorier of ENS Cachan Brittany has developed and released Darshan-Ruby, an object-oriented extension to simplify Darshan log analysis in Ruby.  Darshan-Ruby uses the Darshan logutils API to efficiently access Darshan data without intermediate conversion into text format.
Please visit the Darshan-Ruby web site for more information.

Uncategorized

"Production I/O Characterization on the Cray XE6" to be presented at CUG 2013

April 11th, 2013
Comments Off

We will be presenting a new paper entitled “Production I/O Characterization on the Cray XE6″ at this year’s CUG meeting, May 6-9, in Napa CA.  This presentation will include our experiences in evaluating and deploying Darshan on the Hopper system at NERSC.  We will also use data collected on Hopper to highlight example metrics to identify applications that can benefit from additional tuning.

Uncategorized