Overview
Tensorion is an optimizer designed for machine learning models, introduced as a tensor-aware generalization of the Muon optimizer. It extends Muon's constrained optimization framework from matrices to higher-order tensors, aiming to leverage the multilinear weight structure inherent in various modern machine learning architectures. The core of Tensorion involves a linear minimization oracle (LMO) applied over a specific tensor norm ball.
Research Context
Many first-order optimizers, such as Adam, commonly treat each parameter block as an unstructured vector. This approach can overlook the multilinear weight structure present in contemporary machine learning models, potentially limiting optimization efficiency. Previous research has suggested that exploiting matrix structure can enhance optimization dynamics. The Muon optimizer, for instance, performs steepest descent under a spectral norm constraint, providing an example of how structural awareness can be beneficial.
Approach
Tensorion was developed to address the multilinear weight structure by extending the constrained optimization principles of Muon to higher-order tensors. The design of Tensorion centers on a linear minimization oracle (LMO) operating over a tensor norm ball. The selection of this tensor norm is critical, balancing two objectives: providing a tight bound for the tensor spectral norm while ensuring the LMO remains computationally tractable. The tractability of the LMO in Tensorion is achieved by reducing it to operations on adaptively selected unfolding matrices.
A specific characteristic of Tensorion is its behavior when applied to order-2 tensors. In this scenario, Tensorion is designed to recover the Muon optimizer exactly, indicating a direct generalization from matrix-based to tensor-based optimization within the established framework.
Findings
Experiments were conducted using Tensorion on tensor-based computer vision problems. These evaluations suggested that Tensorion can offer improved convergence behavior. Additionally, the experiments indicated that Tensorion provides more stable gradient updates when compared against both Adam-based optimizers and existing tensor-aware baselines in the evaluated settings.
Why This Matters
The explicit consideration and exploitation of the multilinear weight structure in machine learning models, as attempted by Tensorion, can improve optimization dynamics, leading to enhanced performance and stability in certain applications.