We have performed our trials on sixty-eight variants of seventeen different applications, which constitute version 2.0 of the COPS set. The implementations are written in the AMPL modeling language for use with the AMPL (version 20000906) interfaces to nonlinearly constrained optimization solvers of interest to us. The solvers include LANCELOT (AMPL driver 19990513), LOQO 5.03 (20000528), MINOS 5.5 (19981015), and SNOPT 5.3-4 (19981124).
We have devised a script for running a problem on each solver successively, so as to minimize the effect of fluctuation in the machine load. The script tracks the wall-clock time from the start of the solve, killing any process that runs for more than 3,600 seconds, which we declare unsuccessful. We cycle through all problem variants, recording the wall-clock time as well as the combination of AMPL system time (to interpret the model and compute varying amounts of derivative information required by each solver) and solver time. We consider the times returned by AMPL definitive, but we initially record the wall-clock times to check for discrepancies in the solvers' methods of calculating execution time. We include no problem results for which the AMPL time and the wall-clock time differ by more than ten percent. To further ensure consistency, we have verified that the AMPL time results we present could be reproduced to within ten percent accuracy. All computations were done on a SparcULTRA2 running Solaris 7.
Once all the runs have completed, a parser searches the output files for key text patterns indicating whether the solver completed successfully. The script then gathers the data we need into tables and other files for later calculations.
The AMPL interfaces to these solvers provide numerous options. We set options for each solver and execute our final complete runs with the same options for all problems. The options involve setting the output level so that we can gather the data we want, increasing the iteration limits as much as allowed, and increasing the super-basics limits for MINOS and SNOPT to 5000. None of the failures we record in the final trials include any solver error messages about having violated these limits.
We realize that testing optimization software is a notoriously difficult problem and that there may be objections to the testing presented in this report. For example, performance of a particular solver may improve significantly if non-default options are given. Another objection is that we only use one starting point per problem and that the performance of a solver may be sensitive to the choice of starting point. We also have used the default stopping criteria of the solvers. This choice may bias results but should not affect comparisons that rely on large time differences. In spite of these objections, we feel that it is essential that we provide some indication of the performance of optimization solvers on interesting problems. This report is an effort in this direction.