Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone linesearch algorithms

We propose ZeroFPR, a nonmonotone linesearch algorithm for minimizing the sumof two nonconvex functions, one of which is smooth and the other possibly nonsmooth. ZeroFPRis the first algorithm that, despite being fit for fully nonconvex problems and requiring only theblack-box oracle of forward-backward splitting (FBS)|namely evaluations of the gradient of thesmooth term and of the proximity operator of the nonsmooth one|achieves superlinear convergencerates under mild assumptions at the limit point when the linesearch directions satisfy a Dennis{Moŕecondition, and we show that this is the case for Broyden's quasi-Newton directions. Our approach isbased on the forward-backward envelope (FBE), an exact and strictly continuous penalty functionfor the original cost. Extending previous results we show that, despite being nonsmooth for fullynonconvex problems, the FBE still enjoys favorable first-and second-order properties which are keyfor the convergence results of ZeroFPR. Our theoretical results are backed up by promising numericalsimulations. On large-scale problems, by computing linesearch directions using limited-memoryquasi-Newton updates our algorithm greatly outperforms FBS and its accelerated variant (AFBS).