Automatic differentiation
Divided-difference approximation
MM5, which is developed jointly by the Penn State University (PSU) Meteorology Department and the National Center for Atmospheric Research (NCAR), is a 3-D primitive-equations mesoscale weather model. Sensitivity analysis techniques are employed in atmospheric modeling, for example, to develop a measure of the reliability of a forecast or to assess to what extent a linearization of the model predicts the overall model behavior.
To validate the sensitivity-enhanced MM5 code generated by ADIFOR, we can use automatic differentiation (AD) to produce a Tangent Linear Model (TLM) from MM5 by applying first-order perturbation theory. We then compare the sensitivities predicted by the TLM to divided-difference estimates obtained by running MM5 with small but finite perturbations about the base state. Divided-difference (DD) approximations to derivatives suffer from well-known accuracy limitations when computed using floating-point arithmetic. Even when using the optimal stepsize, the DD generally has an accuracy that is much less than the machine precision; usually only the leading few decimal-places are significant. Furthermore, the optimal stepsize must generally be determined by trial and error. By contrast, automatic differentiation yields a derivative whose accuracy is almost always comparable to that of the function being computed.
Automatic differentiation
Divided-difference approximation
Numerical approximation of a derivative by a divided difference (DD) suffers from a well-known limitation that a DD is much less accurate than the function it was computed from. Consider, for example, the first-order divided-difference approximation ( f(x+h) - f(x) ) / h. There is an optimal value for the stepsize h, since if it is too large, truncation error will dominate the result, whereas if it is too small, roundoff error and cancellation of significant digits will dominate. However, the optimum stepsize is seldom known in advance, because it depends on the second derivative of f, which is unlikely to be available if the first derivative is unknown; hence, the optimal h must be determined by trial and error.
Even for the optimal choice of h, the DD typically only has a relative accuracy of about the square root of machine precision; this yields only 3.6 significant decimals on a 32-bit machine, or 8.4 on a 64-bit machine. Using a higher-order divided difference formula will cost more function evaluations, with no guarantee of better results. Higher order does not necessarily yield higher accuracy unless the optimum stepsize has been chosen --- and again, the optimum stepsize depends on the unknown value of a derivative of higher order than that of the difference formula. Finally, if the function f is extremely complicated, accumulated floating-point errors will erode the accuracy of the DD even more.
By contrast, automatic differentiation yields a derivative whose accuracy is almost always comparable to that of the function itself.
Our current study provided a dramatic demonstration of this fact. By comparing a plot of the temperature sensitivity as computed at timestep 12 (60 mins) by AD to that computed by DD (using a second-order forward-difference formula involving the unperturbed, one-degree perturbed, and two-degree perturbed predictions), we see that the AD and DD results agree closely in appearance. However, the DD approximation suffers from substantial loss of accuracy because all but the last three decimal digits of the mantissas cancel; this manifests itself in the mottled appearance of the DD frame. Choosing a smaller amplitude perturbation exacerbates this problem. (Note that one- and two-degree perturbations are rather large --- on the order of 0.3% and 0.6% of the data, respectively.)
Temperature sensitivities
Pressure sensitivities
More information is available in this report.
This work was supported by the Mathematical, Information, and Computational Sciences Division subprogram of the Office of Computational and Technology Research, U.S. Department of Energy, under Contract W-31-109-Eng-38, and by the U.S. Air Force.
For more information, contact autodiff@mcs.anl.gov.