Argonne National Laboratory

Analyzing How We Do Analysis and Consume Data, Results from the SciDAC-Data Project

TitleAnalyzing How We Do Analysis and Consume Data, Results from the SciDAC-Data Project
Publication TypeJournal Article
Year of Publication2017
AuthorsDing, P, Aliaga, L, Mubarak, M, Tsaris, A, Norman, A, Lyon, A, Ross, R
JournalJournal of Physics: Conference Series
IssueTrack 7: Middleware, Monitoring and Accounting
AbstractOne of the main goals of the Dept. of Energy funded SciDAC-Data project is to analyze the more than 410,000 high energy physics datasets that have been collected, generated and de ned over the past two decades by experiments using the Fermilab storage facilities. These datasets have been used as the input to over 5.6 million recorded analysis projects, for which detailed analytics have been gathered. The analytics and meta information for these datasets and analysis projects are being combined with knowledge of their part of the HEP analysis chains for major experiments to understand how modern computing and data delivery is being used. We present the rst results of this project, which examine in detail how the CDF, D0, NOvA, MINERvA and MicroBooNE experiments have organized, classi ed and consumed petascale datasets to produce their physics results. The results include analysis of the correlations in dataset/ le overlap, data usage patterns, data popularity, dataset dependency and temporary dataset consumption. The results provide critical insight into how workflows and data delivery schemes can be combined with di erent caching strategies to more eciently perform the work required to mine these large HEP data volumes and to understand the physics analysis requirements for the next generation of HEP computing facilities. In particular we present a detailed analysis of the NOvA data organization and consumption model corresponding to their rst and second oscillation results (2014-2016) and the rst look at the analysis of the Tevatron Run II experiments. We present statistical distributions for the characterization of these data and data driven models describing their consumption.