Overview
This study investigates the mechanism by which deep networks perform hierarchical feature learning. It specifically examines the structures of intermediate features within these networks. The research proposes that linear layers can mimic the roles of deep layers in nonlinear networks for feature learning. This investigation leads to a quantitative characterization of feature evolution in hierarchical representations, particularly within deep linear networks.
Research Context
Deep learning has demonstrated effectiveness in learning features from raw data over the past decade. However, the process by which deep networks achieve hierarchical feature learning across their layers has remained an open question. This work attempts to address this by focusing on how deep networks transform input data into output through the features generated at each layer after training. The focus is on multi-class classification problems.
Approach
The research initially observed empirically that linear layers exhibit a similar function to deep layers in nonlinear networks regarding feature learning. Motivated by this, the study explores deep linear networks. To understand how these networks process information, the research defines two metrics: one to measure within-class compression and another to measure between-class discrimination of intermediate features. These metrics are then used for theoretical analysis.
The theoretical analysis considers conditions where input data is nearly orthogonal and network weights are minimum-norm, balanced, and approximately low-rank. The evolution of features under these conditions is then examined.
Empirical validation was conducted through experiments designed to numerical test the theoretical findings. The study also explored whether similar patterns observed in linear networks extend to deep nonlinear networks. Furthermore, the practical implications of these findings were demonstrated in the context of transfer learning.
Findings
Through theoretical analysis, the study showed that when input data is nearly orthogonal and network weights are minimum-norm, balanced, and approximate low-rank, the evolution of features in a linear network follows a specific, quantifiable pattern:
- Each layer of the linear network progressively compresses within-class features at a geometric rate.
- Each layer discriminates between-class features at a linear rate.
These rates are with respect to the number of layers that data have traversed.
Empirical experiments numerically validated these theoretical results. The experiments also indicated a similar pattern of feature evolution in deep nonlinear networks, aligning with prior empirical studies. The research offers a quantitative characterization of feature evolution in hierarchical representations within deep linear networks.
Potential Applications
The research demonstrated practical implications of its results in the domain of transfer learning.