Overview
This work proposes novel scoring functions for softmax attention, leveraging computationally efficient structured matrices. The introduced methods, Block Tensor-Train (BTT) and contiguous Multi-Level Low Rank (MLR) matrices, aim to address limitations in standard attention mechanisms, particularly concerning information loss with high-dimensional inputs and the absence of distance-dependent computational biases.
Research Context
Attention mechanisms, foundational in various transformer architectures, utilize a scoring function that transforms inputs into low-dimensional queries and keys before performing dot products. A noted characteristic of this standard approach is that low-dimensional projection can lead to information loss, particularly when processing intrinsically high-dimensional inputs. Additionally, standard attention applies the same scoring function uniformly across all input pairs, meaning it does not inherently impose a computational bias related to the distance between neighboring tokens in a sequence.
The research seeks to refine attention's inductive biases by customizing its scoring function. This customization aims to integrate either full-rank capabilities or distance-dependent computational biases, directly addressing the identified shortcomings of uniform scoring and potential information degradation for specific input types.
Approach
The core of the proposed approach involves implementing new scoring functions, diverging from the standard dot product between low-dimensional queries and keys. These new functions are constructed using computationally efficient structured matrices:
- Block Tensor-Train (BTT) matrices: These matrices are designed to imbue attention with specific structured properties.
- Contiguous Multi-Level Low Rank (MLR) matrices: These matrices are specifically structured to handle sequential data and introduce distance-dependent biases.
Both BTT and MLR matrices are classified as high-rank structured matrices. The underlying principle is that these structured matrices can be designed to encode specific inductive biases, such as full-rank representations or biases sensitive to distances between tokens, directly into the attention scoring mechanism. The efficiency of these matrices is a key consideration, ensuring that the enhanced capabilities do not lead to prohibitive computational costs.
Findings
The evaluation of the proposed scoring functions yielded several findings across different tasks:
- In-context regression with high-dimensional inputs: On tasks involving in-context regression with intrinsically high-dimensional inputs, the proposed scoring functions (using BTT and MLR) exhibited superior performance compared to standard attention. This advantage was observed across various fixed compute budgets, suggesting improved efficiency or effectiveness for this input type.
- Language modeling: For language modeling, a domain known to exhibit locality patterns, the MLR-based attention method achieved improved scaling laws. This improvement was noted when compared against both standard attention and existing variants of sliding window attention, suggesting better performance characteristics as models scale.
- Encoding inductive biases: Both BTT and MLR matrices were demonstrated to belong to a broader family of efficient structured matrices. This family is capable of encoding either full-rank inductive biases or distance-dependent compute biases, directly addressing the identified limitations of standard attention.
- Long-range time-series forecasting: Preliminary results indicated that MLR attention showed promising performance in tasks involving long-range time-series forecasting.
Why This Matters
The development of custom inductive biases for softmax attention through structured matrices offers a pathway to mitigate information loss when processing high-dimensional inputs and introduces mechanisms for distance-dependent computation. This could lead to more efficient and effective attention models across various computational tasks, particularly those benefiting from localized or structured interactions within input sequences.