Overview
LINet (Linear Integration Network) is presented as a Multi-Stream Neural Network (MSNN) designed for RGB-D scene classification. This architecture introduces a Continuous Linear Integration approach to cross-modal learning, differentiating it from conventional multi-modal fusion methods. The core mechanism involves a novel Linear Integration Convolution (LIConv2d) operator that facilitates continuous signal integration across three parallel streams: RGB, depth, and an integrating stream.
The integration stream's function is to receive and combine raw filtered signals from both the RGB and depth modality streams prior to their nonlinear activation threshold. This design was conceptualized with inspiration from somatic integration observed before neuronal firing decisions. The LINet architecture, particularly its continuous integration mechanism, required specific initialization strategies and training curricula to address identified pathologies.
Research Context
Current multi-modal architectural designs for feature fusion generally fall into categories such as early, late, or hybrid fusion, including methods based on attention. These approaches typically treat feature fusion as a discrete event. Early fusion is characterized by intertwining representations prematurely, while late fusion isolates modalities until the final output layer. Hybrid or attention-based methods necessitate an architectural estimation process to determine the placement of intermediate fusion blocks within the network structure. LINet aims to address what is described as a 'structural compromise' in these existing approaches by enabling cross-modal learning at every layer continuously.
Approach
LINet maintains three dedicated parallel streams: one for RGB data, one for depth data, and an integration stream. The key component, the LIConv2d operator, is implemented to enable continuous cross-modal learning. This operator allows the integration stream to combine raw filtered signals directly from both modality streams before the application of the nonlinear activation. This process is conceptually analogous to somatic integration observed in biological neurons, preceding the neuronal firing decision.
A critical challenge identified during the implementation of continuous integration was an 'initialization pathology.' Specifically, applying Kaiming initialization to the bridging weights within the LIConv2d operator was found to scramble gradients before they could effectively reach the stream backbones. This issue manifested as a failure mode that superficially resembled overfitting but was ultimately attributed to corrupted gradient flow. To mitigate this effect, a $1/N$ constant initialization strategy was adopted for the bridging weights, resolving the issue.
Furthermore, the training process for LINet incorporated 'progressive modality dropout.' This curriculum is specifically adapted for continuous fusion architectures. It involves gradually increasing the probability of blanking one or more modalities from zero. This strategy was implemented to prevent 'pathway collapse,' a form of negative co-learning, by compelling the network to develop robust and independentrepresentations within each stream.
Findings
- Trained from scratch on the SUN RGB-D 19-class scene classification benchmark, LINet achieved a mean class accuracy of 45.2% at the ResNet18 scale. This performance level was observed to outperform prior results obtained under similar from-scratch training conditions.
- When leveraging in-domain RGB-D pretraining from the ScanNet dataset, the mean class accuracy of LINet further increased to 49.6%.
Why This Matters
The development of LINet offers an alternative paradigm for multi-modal feature fusion in neural networks, specifically addressing the design choices inherent in discrete fusion strategies. By demonstrating improved performance on standard RGB-D scene classification benchmarks through its continuous integration approach, LINet provides a new architectural framework for processing combined visual and depth information more effectively using convolutional neural networks.