darshan.backend package

The backend package provides implementions for reading the binary representation of the darshan log file. The only current implementation uses CFFI which uses the functions defined in libdarshan-util.so to read the log.

Submodules

darshan.backend.api_def_c module

The api_def_c carries a copy of CFFI compatible headers for libdarshan-util.so. These definitions must match the structure definitions for the associated darshan release.

darshan.backend.api_def_c.load_darshan_header(addins='')

Returns a CFFI compatible header for darshan-utlil as a string.

Returns:

String with a CFFI compatible header for darshan-util.

darshan.backend.apmpi module

darshan.backend.apmpi.get_apmpi_defs()

darshan.backend.apmpi.log_get_apmpi_record(log, mod_name, structname, dtype='dict')

darshan.backend.apss module

darshan.backend.apss.get_apss_defs()

darshan.backend.apss.log_get_apss_record(log, mod_name, structname, dtype='dict')

darshan.backend.apxc module

darshan.backend.apxc.get_apxc_defs()

darshan.backend.apxc.log_get_apxc_record(log, mod_name, structname, dtype='dict')

darshan.backend.cffi_backend module

The cfii_backend package will read a darshan log using the functions defined in libdarshan-util.so and is interfaced via the python CFFI module.

darshan.backend.cffi_backend._df_to_rec(rec_dict, mod_name, rec_index_of_interest=None)

Pack the DataFrames-format PyDarshan data back into a C buffer of records that can be consumed by darshan-util C code.

Parameters

rec_dict: dict

Dictionary containing the counter and fcounter dataframes.

mod_name: str

Name of the darshan module.

rec_index_of_interest: int or None

If None, use all records in the dataframe. Otherwise, repack only the the record at the provided integer index.

Returns

buf: Raw char array containing a buffer of record(s) or a single record.

darshan.backend.cffi_backend._log_get_heatmap_record(log)

Returns a dictionary holding a heatmap darshan log record.

Parameters:

log – Handle returned by darshan.open

Returns:

heatmap log record

Return type:

dict

darshan.backend.cffi_backend._log_get_lustre_record(log, dtype='numpy')

Returns a darshan log record for Lustre.

Parameters:

log – handle returned by darshan.open

darshan.backend.cffi_backend._make_generic_record(rbuf, mod_name, dtype='numpy')

Returns a record dictionary for an input record buffer for a given module.

darshan.backend.cffi_backend.accumulate_records(rec_dict, mod_name, nprocs)

Passes a set of records (in pandas format) to the Darshan accumulator interface, and returns the corresponding derived metrics struct and summary record.

Parameters:

• rec_dict – Dictionary containing the counter and fcounter dataframes.

• mod_name – Name of the Darshan module.

• nprocs – Number of processes participating in accumulation.

Returns:

namedtuple containing derived_metrics (cdata object) and summary_record (dict).

darshan.backend.cffi_backend.counter_names(mod_name, fcnts=False, special='')

Returns a list of available counter names for the module. By default only integer counter names are listed, unless fcnts is set to true in which case only the floating point counter names are listed.

Parameters:

• mod_name (str) – Name of the module to return counter names.

• fcnts (bool) – Switch to request floating point counters instead of integer. (Default: False)

Returns:

Counter names as strings.

Return type:

list

darshan.backend.cffi_backend.fcounter_names(mod_name)

Returns a list of available floating point counter names for the module.

Parameters:

mod_name (str) – Name of the module to return counter names.

Returns:

Available floiting point counter names as strings.

Return type:

list

darshan.backend.cffi_backend.get_lib_version()

Return the version information hardcoded into the shared library.

Parameters:

None –

Returns:

library version number

Return type:

version (str)

darshan.backend.cffi_backend.log_close(log)

Closes the logfile and releases allocated memory.

darshan.backend.cffi_backend.log_get_dxt_record(log, mod_name, reads=True, writes=True, dtype='dict')

Returns a dictionary holding a dxt darshan log record.

Parameters:

• log – Handle returned by darshan.open

• mod_name (str) – Name of the Darshan module

• mod_type (str) – String containing the C type

Returns:

generic log record

Return type:

dict

Example:

The typical darshan log record provides two arrays, on for integer counters and one for floating point counters:

>>> darshan.log_get_dxt_record(log, "DXT_POSIX", "struct dxt_file_record **")
{'rank': 0, 'read_count': 11, 'read_segments': array([...]), ...}

darshan.backend.cffi_backend.log_get_exe(log)

Get details about the executable (path and arguments)

Parameters:

log – handle returned by darshan.open

Returns:

executable path and arguments

Return type:

string

darshan.backend.cffi_backend.log_get_generic_record(log, mod_name, dtype='numpy')

Returns a dictionary holding a generic darshan log record.

Parameters:

• log – Handle returned by darshan.open

• mod_name (str) – Name of the Darshan module

Returns:

generic log record

Return type:

dict

Example:

The typical darshan log record provides two arrays, on for integer counters and one for floating point counters:

>>> darshan.log_get_generic_record(log, "POSIX", "struct darshan_posix_file **")
{'counters': array([...], dtype=int64), 'fcounters': array([...])}

darshan.backend.cffi_backend.log_get_job(log)

Returns a dictionary with information about the current job.

darshan.backend.cffi_backend.log_get_modules(log)

Return a dictionary containing available modules including information about the contents available for each module in the current log.

Parameters:

log – handle returned by darshan.open

Returns:

Modules with additional info for current log.

Return type:

dict

darshan.backend.cffi_backend.log_get_mounts(log)

Returns a list of available mounts recorded for the log.

Parameters:

log – handle returned by darshan.open

darshan.backend.cffi_backend.log_get_name_records(log)

Return a dictionary resovling hash to string (typically a filepath).

Parameters:

• log – handle returned by darshan.open

• hash – hash-value (a number)

Returns:

the name records

Return type:

dict

darshan.backend.cffi_backend.log_get_record(log, mod, dtype='numpy')

Standard entry point fetch records via mod string.

Parameters:

• log – Handle returned by darshan.open

• mod_name (str) – Name of the Darshan module

Returns:

log record of type dtype

darshan.backend.cffi_backend.log_lookup_name_records(log, ids=[])

Resolve a single hash to it’s name record string (typically a filepath).

Parameters:

• log – handle returned by darshan.open

• hash – hash-value (a number)

Returns:

the name records

Return type:

dict

darshan.backend.cffi_backend.log_open(filename)

Opens a darshan logfile.

Parameters:

filename (str) – Path to a darshan log file

Returns:

log handle

darshan.backend.cffi_backend.mod_name_to_idx(mod_name)