Overview
This research investigates a class of weakly convex optimization problems characterized by an objective function comprising two distinct terms: a smooth convex component and a potentially nonsmooth weakly convex component. To address this structure, the study developed and analyzed two optimization algorithms: the Alternating Direction Method of Multipliers (ADMM) and a linearized variant, Linearized ADMM (LADMM). Both methods aim to decompose the original problem into subproblems that are individually tractable, leveraging the specific characteristics of each component of the objective function.
Research Context
The focus is on optimization problems where the objective function $f(x)$ can be expressed as the sum of a smooth convex term $g(x)$ and a weakly convex term $h(x)$ that may be nonsmooth. This specific problem structure, represented as $f(x) = g(x) + h(x)$, informed the design of the algorithmic approaches. The presence of a smooth convex term and a potentially nonsmooth weakly convex term indicates a need for methods that can handle both differentiability and nondifferentiability efficiently, while also accommodating the 'weakly convex' property.
Approach
The core methodology involves a splitting technique based on the ADMM framework. This technique was designed to decouple the minimization of the smooth convex term and the weakly convex term into separate subproblems. This decoupling strategy aims to make each individual minimization step more manageable.
- ADMM Scheme: The primary ADMM method was developed to exploit the problem's sum structure.
- Linearized ADMM (LADMM) Scheme: Recognizing that the update associated with the smooth term might necessitate an inner iterative solver, a linearized version of ADMM (LADMM) was introduced. The LADMM scheme addresses this by linearizing the smooth term, leading to a one-step update that is described as inexpensive.
The theoretical analysis established the subsequence convergence of both the ADMM and LADMM methods. This convergence was demonstrated to directional stationary solutions. For the specific class of weakly convex problems under consideration, these directional stationary solutions were found to be equivalent to critical points and Clarke stationary solutions.
Findings
The study yielded several key findings regarding the proposed ADMM and LADMM methods:
- Convergence: Under specified mild conditions, both ADMM and LADMM methods demonstrated subsequence convergence. The convergence target was identified as directional stationary solutions.
- Equivalence of Solutions: For the weakly convex problems studied, the identified directional stationary solutions were shown to be equivalent to critical points and Clarke stationary solutions.
- Computational Efficiency: Numerical experiments indicated that the proposed ADMM and LADMM approaches were computationally efficient.
- Solution Quality: The numerical experiments also suggested that the solutions produced by these methods were of comparable quality to those obtained from baseline methods.
- Experimental Scope: Numerical evaluations were performed on two low-dimensional test functions and a high-dimensional logarithmic regularized logistic regression model.
Why This Matters
The development of ADMM and LADMM for weakly convex problems provides algorithmic tools for a specific class of optimization challenges. The established convergence properties and observed computational efficiency suggest their utility for problems with the identified objective function structure. The equivalence of directional stationary solutions to critical and Clarke stationary points clarifies the nature of the solutions obtained.