This is a library for unconstrained minimization of smooth functions with a large number of variables. I wrote this to get a better grasp of nonlinear optimization. I used the book ny Nocedal and Wright [1] as a reference.

• Newton's method
  • Bunch-Kaufman-Parlett factorization and block-diagonal modification. This is a robust method of dealing with nonconvex functions.
  • The sparse solver uses repeated diagonal modification of the hessian for nonconvex problems. This simple method seems to work well, but can require several Cholesky factorizations per iteration and is not as robust as B-K-P.
• Sparse Cholesky factorization using Eigen (included) if the problem is large and sparse.
• L-BFGS.
• Nelder-Mead for nondifferentiable problems.
• Automatic differentiation to compute gradient and hessian using FADBAD++ (included).
• Multi-threaded using OpenMP.
• The interface is very easy to use, while still allowing very high performance. Generic lambdas (C++14) can be used if the compiler supports it.

####Experimental features This repository also contains some experimental features. These features are not ready for production and do not have very extensive test coverage. They may change or be removed in the future.

• Wolfe line search.
• Global optimization with interval arithmetic.
• Constrained optimization (augmented lagrangian).

####To Do

• Sparse block-diagonal factorization.

Everything needed to compile the library, examples and tests using CMake should be included. All tests pass with the following compilers:

• Visual Studio 2013
• GCC 4.8 (Cygwin)
• GCC 4.7 (Ubuntu)
• Clang 3.3 (Ubuntu) Earlier compilers will probably not work.

You can check travis.yml for the commands used to build the library and run all tests on Ubuntu. It is even easier on Windows. The status of the automatic builds using gcc and Clang is .

The solver comes with extensive benchmarks and tests.

• The NIST collection of non-linear least-squares problems. http://www.itl.nist.gov/div898/strd/nls/nls_main.shtml
• Test functions from More et al. [2].
• TEST_OPT http://people.sc.fsu.edu/~jburkardt/m_src/test_opt/test_opt.html

The Newton solver and the L-BFGS solver pass all of these tests. The NIST collection was very challenging and required block-diagonal robust factorization and handling of numerically hard problem instances. Note that non-linear least-squares problems have a special structure and are best solved with custom code, for example Ceres Solver.

The interface supports generic lambdas:

auto lambda =
	[](auto x, auto y)
	{
		// The Rosenbrock function.
		auto d0 =  y[0] - x[0]*x[0];
		auto d1 =  1 - x[0];
		return 100 * d0*d0 + d1*d1;
	};

// The lambda function will be copied and
// automatically differentiated. The derivatives
// are computed using templates, not numerically.
//
// No need to derive or compute derivatives
// manually!
auto term = make_differentiable<1, 1>(lambda);

double x=0, y=0;
function.add_term(term, &x, &y);

If you have the November 2013 CTP installed for Visual Studio 2013, this new interface is unit tested in test_generic_lambdas.cpp.

1. Nocedal and Wright, Numerical Optimization, Springer, 2006.
2. Jorge J. More, Burton S. Garbow and Kenneth E. Hillstrom, Testing unconstrained optimization software, Transactions on Mathematical Software 7(1):17-41, 1981.