User Level Memory Map (UMap) is a memory map approach to incorporating
NVRAM into complex memory hierarchy. It replaces
for out-of-core data. By exploiting large virtual address spaces, data
sets can be accessed directly as if in memory; this is particularly well
suited to analyzing large observational and simulation data. Thanks to its
user-space design, UMap is flexible and easy to use.
Migration of pages between far and near memory is the responsibility of the
UMap handler. It uses the low-overhead
enabling applications to handle page faults in user space. The UMap handler
receives a page fault notification as a message and loads the requested
page. Since it runs in the application context, the handler can be specific to
an application(-class). The page size (multiple of system page) and the page
buffer size are customizable, and the data can be loaded from multiple
files. Support for loading from distributed data sets—over the
network—is under development.
The second figure illustrates an application that uses UMap. The application takes imaging data sets of the sky generated by an optical camera and uses them to detect Near-Earth Asteroids (NEAs). Our transient detection algorithm computes medians along likely NEA trajectories to track asteroid motion in imagery over time. UMap combines many 2-D tiled images to form a 3-D volume for the search algorithm.
Another class of applications we studied is out-of-core graph algorithms that
generate an edge or adjacency list representation of graphs from raw data
(e.g., Wikimedia Commons edits) and traverse the graphs to compute features
of interest for community detection.
Maya Gokhale, Marty McFadden, Roger Pearce, Eric Green, Keita Iwabuchi (LLNL); Kai Wu, Dong Li (UC Merced and LLNL)