PyDarshan Documentation

Python utilities to interact with Darshan log records of HPC applications. PyDarshan requires darshan-utils version 3.3 or higher to be installed.

Features

• Darshan Report Object for common interactive analysis tasks

• Low-level CFFI bindings for efficient access to darshan log files

• Plots typically found in the darshan reports (matplotlib)

• Bundled with darshan-utils while allowing site’s darshan-utils to take precedence

Usage

For examples and Jupyter notebooks to get started with PyDarshan make sure to check out the examples subdirectory.

A brief examples showing some of the basic functionality is the following:

import darshan

# Open darshan log
with darshan.DarshanReport('example.darshan', read_all=False) as report:

    # Load some report data
    report.mod_read_all_records('POSIX')
    report.mod_read_all_records('MPI-IO')
    # or fetch all
    report.read_all_generic_records()

    # ...
    # Generate summaries for currently loaded data
    # Note: aggregations are still experimental and have to be activated:
    darshan.enable_experimental()
    report.summarize()

Installation

To install in most cases the following will work:

pip install --user darshan

For alternative installation instructions and installation from source refer to <docs/install.rst>

Testing

Targets for various tests are included in the makefile. To run the normal test suite use:

make test

Or to test against different version of Python using Tox:

make test-all

Coverage tests can be performed using:

make coverage

Conformance to PEPs can be tested using flake8 via:

make lint

Documentation

Documentation for the python bindings is generated seperatedly from the darshan-utils C library in the interest of using Sphinx. After installing the developement requirements using pip install -r requirements_dev.txt the documentation can be build using make as follows:

pip install -r requirements_dev.txt
make docs

File List

• darshan::

  core darshan python module code

• devel::

  scripts for building python wheel

• docs::

  markdown documentation used by sphinx to auto-generate HTML RTD style doc

• examples::

  Jupyter notebooks showing PyDarshan usage with log files

• tests::

  PyDarshan specific test cases

• requirements.txt::

  pip requirement file for minimum set of depednencies

• requirements_dev.txt::

  pip requirement file for depednencies needed to run development tools

• setup.py::

  python file for building/generating PyDarshan package

• setup.cfg::

  input for setup.py

• MANIFEST.in::

  input files for setup.py package

• tox.ini::

  input for tox which runs the automated testing