Overview
TS2Vec, an established self-supervised architecture introduced in 2022 for time series classification, was originally engineered to undergo self-supervised pre-training and linear probing separately on every target dataset. Researchers demonstrate that the model can be converted into a general foundation model—termed TS2Vec-FM—that is pre-trained only once and evaluated with frozen weights across multiple benchmark suites.
Research Context
In time series classification, recent developments have increasingly focused on foundation models capable of reusing a single encoder across different datasets. While conventional foundation models often incorporate attention mechanisms, patch tokenizers, or intermediate-layer feature extraction, original TS2Vec relies on a compact convolutional architecture. This architecture natively handles variable-length temporal sequences without requiring resampling and processes temporal samples directly. The study examines whether these structural properties can be preserved while eliminating the requirement to retrain the encoder per dataset.
Approach
The transition from dataset-specific TS2Vec to TS2Vec-FM relies on three specific modifications:
- Synthetic Corpus Pre-training: Pre-training the model once on a synthetic corpus rather than on target datasets individually.
- Dual-View Input Stems: Passing two distinct perspectives of the input sequence into the encoder via two small two-layer multilayer perceptron (MLP) stems: an instance-normalized shape view and a raw amplitude view.
- Encoder Distillation: Training specialist encoders for each view and subsequently distilling them into a single, unified encoder.
The resulting TS2Vec-FM network comprises a 1.6M-parameter dilated convolutional backbone coupled with the two-layer MLP stem, bypassing the need for attention layers, patch tokenization, or intermediate-layer extraction.
Findings
Evaluated under linear probing with its encoder weights held frozen, TS2Vec-FM achieved:
- 83.3% accuracy on the UCR-128 benchmark.
- 75.3% accuracy on the UEA-27 benchmark.
- Performance exceeding that of standard TS2Vec trained separately on each individual target dataset.
- Accuracy within 0.3 percentage points of MantisV2, the current state of the art in the domain.