User Level Memory Map (UMap) is a memory map approach to incorporating NVRAM into complex memory hierarchy. It replaces mmap for out-of-core data. By exploiting large virtual address spaces, data sets can be accessed directly as if in memory; this is particularly well suited to analyzing large observational and simulation data. Thanks to its user-space design, UMap is flexible and easy to use.

UMap architecture Migration of pages between far and near memory is the responsibility of the UMap handler. It uses the low-overhead userfaultfd protocol, enabling applications to handle page faults in user space. The UMap handler receives a page fault notification as a message and loads the requested page. Since it runs in the application context, the handler can be specific to an application(-class). The page size (multiple of system page) and the page buffer size are customizable, and the data can be loaded from multiple files. Support for loading from distributed data sets—over the network—is under development.

UMap application The second figure illustrates an application that uses UMap. The application takes imaging data sets of the sky generated by an optical camera and uses them to detect Near-Earth Asteroids (NEAs). Our transient detection algorithm computes medians along likely NEA trajectories to track asteroid motion in imagery over time. UMap combines many 2-D tiled images to form a 3-D volume for the search algorithm.

Another class of applications we studied is out-of-core graph algorithms that generate an edge or adjacency list representation of graphs from raw data (e.g., Wikimedia Commons edits) and traverse the graphs to compute features of interest for community detection.

More information on UMap can be found in its documentation on Read the Docs. For a complete list of UMap-related resources, check the For Developers page.

Maya Gokhale, Ivy Peng, Abhik Sarkar, Roger Pearce, Keita Iwabuchi (LLNL); Kai Wu, Dong Li (UC Merced and LLNL)