darshan package

PyDarshan provides direct log access for reading binary Darshan logs. PyDarshan also provides a suite of analysis utilities.

darshan.enable_experimental(verbose=False)

Enable experimental features such as aggregation methods for reports.

Parameters:

verbose (bool) – Display log of enabled features. (Default: True)

Subpackages

• darshan.backend package
  • Submodules
  • darshan.backend.api_def_c module
    • load_darshan_header()
  • darshan.backend.cffi_backend module
    • _df_to_rec()
    • _log_get_heatmap_record()
    • _log_get_lustre_record()
    • _make_generic_record()
    • accumulate_records()
    • counter_names()
    • fcounter_names()
    • get_lib_version()
    • log_close()
    • log_get_dxt_record()
    • log_get_exe()
    • log_get_generic_record()
    • log_get_job()
    • log_get_modules()
    • log_get_mounts()
    • log_get_name_records()
    • log_get_record()
    • log_lookup_name_records()
    • log_open()
    • mod_name_to_idx()
• darshan.cli package
  • CustomHelpFormatter
  • discover_subcommands()
  • main()
  • Submodules
  • darshan.cli.info module
    • main()
    • setup_parser()
  • darshan.cli.name_records module
    • main()
    • setup_parser()
  • darshan.cli.summary module
  • darshan.cli.to_json module
    • main()
    • setup_parser()
• darshan.datatypes package
  • Submodules
  • darshan.datatypes.heatmap module
    • Heatmap
      • Heatmap.add_record()
      • Heatmap.info()
      • Heatmap.to_df()
• darshan.examples package
  • Subpackages
    • darshan.examples.example_logs package
      • installed_path()
• darshan.experimental package
  • Subpackages
    • darshan.experimental.aggregators package
      • Submodules
      • darshan.experimental.aggregators.agg_ioops module
      • darshan.experimental.aggregators.create_dxttimeline module
      • darshan.experimental.aggregators.create_sankey module
      • darshan.experimental.aggregators.create_time_summary module
      • darshan.experimental.aggregators.create_timeline module
      • darshan.experimental.aggregators.mod_agg_iohist module
      • darshan.experimental.aggregators.name_records_summary module
      • darshan.experimental.aggregators.print_module_records module
      • darshan.experimental.aggregators.records_as_dict module
      • darshan.experimental.aggregators.summarize module
    • darshan.experimental.plots package
      • Submodules
      • darshan.experimental.plots.data_access_by_filesystem module
      • darshan.experimental.plots.heatmap_handling module
      • darshan.experimental.plots.plot_access_histogram module
      • darshan.experimental.plots.plot_common_access_table module
      • darshan.experimental.plots.plot_dxt_heatmap module
      • darshan.experimental.plots.plot_dxt_heatmap2 module
      • darshan.experimental.plots.plot_io_cost module
      • darshan.experimental.plots.plot_opcounts module
      • darshan.experimental.plots.plot_posix_access_pattern module
• darshan.tests package
  • Subpackages
    • darshan.tests.input package
      • installed_path()
  • Submodules
  • darshan.tests.conftest module
    • pytest_configure()
  • darshan.tests.test_cffi_misc module
    • test_df_to_rec()
    • test_file_hash_type()
    • test_get_lib_version()
    • test_log_get_generic_record()
    • test_reverse_record_array()
  • darshan.tests.test_data_access_by_filesystem module
  • darshan.tests.test_error module
    • test_cannotopen()
  • darshan.tests.test_heatmap_handling module
    • dict_list()
    • dict_list_no_writes()
    • test_get_aggregate_data()
    • test_get_heatmap_df()
    • test_get_rd_wr_dfs()
    • test_get_rd_wr_dfs_no_write()
    • test_get_single_df_dict()
  • darshan.tests.test_lib_accum module
  • darshan.tests.test_log_utils module
    • test_failure_bad_logname()
    • test_retrieve_both_repos()
  • darshan.tests.test_moddxt module
    • test_dxt_records()
  • darshan.tests.test_modmpiio module
    • response()
    • test_counters()
    • test_fcounters()
    • test_ishouldrun()
    • test_repeated_access()
  • darshan.tests.test_modposix module
    • init()
    • test_counters()
    • test_fcounters()
    • test_ishouldrun()
    • test_repeated_access()
  • darshan.tests.test_modstdio module
    • response()
    • test_counters()
    • test_fcounters()
    • test_ishouldrun()
    • test_repeated_access()
  • darshan.tests.test_plot_common_access_table module
    • test_collapse_access_cols()
    • test_common_access_table()
    • test_get_access_count_df()
    • test_misc_funcs()
  • darshan.tests.test_plot_dxt_heatmap module
    • jointgrid()
    • test_adjust_for_colorbar()
    • test_get_y_axis_tick_labels()
    • test_get_y_axis_ticks()
    • test_plot_heatmap()
    • test_remove_marginal_graph_ticks_and_labels()
    • test_set_x_axis_ticks_and_labels()
    • test_set_y_axis_ticks_and_labels()
  • darshan.tests.test_plot_exp_common module
    • test_bar_heights()
    • test_xticks_and_labels()
  • darshan.tests.test_plot_io_cost module
    • test_combine_hdf5_modules()
    • test_get_by_avg_series()
    • test_get_io_cost_df()
    • test_issue_590()
    • test_plot_io_cost_x_ticks_and_labels()
    • test_plot_io_cost_y_ticks_and_labels()
    • test_plot_io_cost_ylims()
  • darshan.tests.test_report module
  • darshan.tests.test_report_copy module
  • darshan.tests.test_summary module

Submodules

darshan.discover_darshan module

Auxiliary to discover a darshan-util installation.

exception darshan.discover_darshan.DarshanVersionError(target_version, provided_version, msg='Feature')

Bases: NotImplementedError

Raised when using a feature which is not provided by libdarshanutil.

darshan.discover_darshan.check_version(ffi=None, libdutil=None)

Get version from shared library or pkg-config and return info.

Returns:

Path to a darshan-util installation.

darshan.discover_darshan.discover_darshan_pkgconfig()

Discovers an existing darshan-util installation and returns the appropriate path to a shared object for use with Python’s CFFI.

Returns:

Path to a darshan-util installation.

darshan.discover_darshan.discover_darshan_pyinstaller()

Discovers darshan-util if installed as as part of a pyinstaller bundle.

Returns:

Path to a darshan-util installation.

darshan.discover_darshan.discover_darshan_shutil()

Discovers an existing darshan-util installation and returns the appropriate path to a shared object for use with Python’s CFFI.

Returns:

Path to a darshan-util installation.

darshan.discover_darshan.discover_darshan_wheel()

Discovers darshan-util if installed as as part of the wheel.

Returns:

Path to a darshan-util installation.

darshan.discover_darshan.find_utils(ffi, libdutil)

Try different methods to discover darshan-util:

Precedence: 1) Try if the current environment allows dlopen to load libdarshan-util 2) Try if darshan-parser is exposed via PATH, and attempt loading relative to it. 3) Try if darshan is exposed via pkgconfig 4) Fallback on binary distributed along with Python package

Parameters:

• ffi – existing ffi instance to use

• libdutil – reference to libdutil to populate

darshan.discover_darshan.load_darshan_header()

Returns a CFFI compatible header for darshan-utlil as a string.

Returns:

String with a CFFI compatible header for darshan-util.

darshan.log_utils module

Module for log handling utilities.

darshan.log_utils._locate_log(filename: str, project: str) → str | None

Locates a log in a project.

darshan.log_utils.get_log_path(filename: str) → str

Utility function for locating logs either locally or in the darshan-logs repo.

Parameters

filename: filename of Darshan log to locate in PyDarshan or darshan-logs repo.

Returns

log_path: absolute path to the darshan log matching the input filename.

Raises

FileNotFoundError: if a log cannot be found that matches the input filename.

Notes

If used in the context of a pytest run, pytest.skip() will be used if both A) a local log cannot be found, and B) the darshan-logs repo is unavailable.

This function should never be used in a pytest decorator/ parametrization mark. While it is possible for the function to retrieve log file paths in such scenarios, the imperative skip is not tolerated at collection time when the logs repo is absent.

darshan.report module

The darshan.repport module provides the DarshanReport class for convienient interaction and aggregation of Darshan logs using Python.

class darshan.report.DarshanRecordCollection(mod=None, report=None)

Bases: MutableSequence

Darshan log records may nest various properties (e.g., DXT, Lustre). As such they can not faithfully represented using only a single Numpy array or a Pandas dataframe.

The DarshanRecordCollection is used as a wrapper to offer users a stable API to DarshanReports and contained records in various popular formats while allowing to optimize memory and internal representations as necessary.

append(val)

S.append(value) – append value to the end of the sequence

info(describe=False, plot=False)

Print information about the record for inspection.

Parameters:

• describe (bool) – show detailed summary and statistics (default: False)

• plot (bool) – show plots for quick value overview for counters and fcounters (default: False)

Returns:

None

insert(key, val)

S.insert(index, value) – insert value before index

to_df(attach='default')

to_dict()

to_json()

to_list()

to_numpy()

class darshan.report.DarshanReport(filename=None, dtype='numpy', start_time=None, end_time=None, automatic_summary=False, read_all=True, lookup_name_records=True)

Bases: object

The DarshanReport class provides a convienient wrapper to access darshan logs, which also caches already fetched information. In addition to that a number of common aggregations can be performed.

__add__(other)

Allow reports to be merged using the addition operator.

__deepcopy__(memo)

Creates a deepcopy of report.

Note

Needed to purge reference to self.log as Cdata can not be pickled: TypeError: can’t pickle _cffi_backend.CData objects

__del__()

Clean up when deleted or garbage collected (e.g., del-statement)

__enter__()

Satisfy API for use with context manager (e.g., with-statement)

__exit__(type, value, traceback)

Cleanup when used by context manager (e.g., with-statement)

__init__(filename=None, dtype='numpy', start_time=None, end_time=None, automatic_summary=False, read_all=True, lookup_name_records=True)

Parameters:

• filename (str) – filename to open (optional)

• dtype (str) – default dtype for internal structures

• automatic_summary (bool) – automatically generate summary after loading

• read_all (bool) – whether to read all records for log

• lookup_name_records (bool) – lookup and update name_records as records are loaded

Returns:

None

_cleanup()

Cleanup when deleting object.

property counters

property heatmaps

info(metadata=False)

Print information about the record for inspection.

Parameters:

metadata (bool) – show detailed metadata (default: False)

Returns:

None

property metadata

mod_read_all_apmpi_records(mod='APMPI', dtype=None, warnings=True)

Reads all APMPI records for provided module.

Parameters:

• mod (str) – Identifier of module to fetch all records

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary

Returns:

None

mod_read_all_apxc_records(mod='APXC', dtype=None, warnings=True)

Reads all APXC records for provided module.

Parameters:

• mod (str) – Identifier of module to fetch all records

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary

Returns:

None

mod_read_all_dxt_records(mod, dtype=None, warnings=True, reads=True, writes=True)

Reads all dxt records for provided module.

Parameters:

• mod (str) – Identifier of module to fetch all records

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary

Returns:

None

mod_read_all_lustre_records(mod='LUSTRE', dtype=None, warnings=True)

Reads all dxt records for provided module.

Parameters:

• mod (str) – Identifier of module to fetch all records

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary

Returns:

None

mod_read_all_records(mod, dtype=None, warnings=True)

Reads all generic records for module

Parameters:

• mod (str) – Identifier of module to fetch all records

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary, ‘pandas’

Returns:

None

mod_records(mod, dtype='numpy', warnings=True)

Return generator for lazy record loading and traversal.

Warning

Can’t be used for now when alternating between different modules. A temporary workaround can be to open the same log multiple times, as this way buffers are not shared between get_record invocations in the lower level library.

Parameters:

• mod (str) – Identifier of module to fetch records for

• dtype (str) – ‘numpy’ for ndarray (default), ‘dict’ for python dictionary

Returns:

None

property modules

open(filename, read_all=False)

Open log file via CFFI backend.

Parameters:

• filename (str) – filename to open (optional)

• read_all (bool) – whether to read all records for log

Returns:

None

read_all(dtype=None)

Read all available records from darshan log and return as dictionary.

Parameters:

None –

Returns:

None

read_all_dxt_records(reads=True, writes=True, dtype=None)

Read all dxt records from darshan log and return as dictionary.

Parameters:

None –

Returns:

None

read_all_generic_records(counters=True, fcounters=True, dtype=None)

Read all generic records from darshan log and return as dictionary.

Parameters:

None –

Returns:

None

read_all_heatmap_records()

Read all heatmap records from darshan log and return as dictionary.

Note

As the module is encoded in a name_record, all heatmap data is read and then exposed through the report.heatmaps property.

Parameters:

None –

Returns:

None

read_metadata(read_all=False)

Read metadata such as the job, the executables and available modules.

Parameters:

None –

Returns:

None

rebase_timestamps(inplace=False, timebase=False)

Updates all records in the report to use timebase (defaults: start_time). This might allow to conserve memory as reports are merged.

Parameters:

• records (dict, list) – records to rebase

• inplace (bool) – weather to merel return a copy or to update records

• timebase (datetime.datetime) – new timebase to use

Returns:

rebased_records (same type as provided to records)

to_dict()

Return dictionary representation of report data.

Parameters:

None –

Returns:

dict

to_json()

Return JSON representation of report data as string.

Parameters:

None –

Returns:

JSON String

update_name_records(mod=None)

Update (and prune unused) name records from resolve table.

First reindexes all used name record identifiers and then queries darshan-utils library to compile filtered list of name records.

Parameters:

None –

Returns:

None

class darshan.report.DarshanReportJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: JSONEncoder

Helper class for JSON serialization if the report contains, for example, numpy or dates records, which are not handled by the default JSON encoder.

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

exception darshan.report.ModuleNotInDarshanLog

Bases: ValueError

Raised when module is not present in Darshan log.