Contrastive Denoising Autoencoder for Perturbation-Invariant Pretrained Language Model Embeddings

Overview

Pre-trained language models (PLMs) have advanced sentence representation learning, yet their generated embeddings exhibit sensitivity to semantic-preserving textual perturbations. Such perturbations include synonym substitution, masking, and word dropout. This work introduces a lightweight Contrastive Denoising Autoencoder (CDAE) designed to refine pre-trained BERT embeddings. The CDAE aims to learn perturbation-invariant representations through a joint optimization of contrastive and reconstruction objectives.

Approach

The proposed framework, CDAE, was developed to enhance the robustness of pre-trained BERT embeddings. The core mechanism involves a joint optimization strategy combining two objectives:

• Contrastive Objective: This component works to pull together representations of semantically similar inputs while pushing apart dissimilar ones. The goal is to learn embeddings that are robust to minor variations.
• Reconstruction Objective: This component ensures that the model can reconstruct the original input from its perturbed version, preserving essential semantic information during the denoising process.

By combining these objectives, CDAE aims to learn representations that maintain high similarity even when the input text undergoes semantic-preserving perturbations. The framework operates as a lightweight refinement layer over existing BERT embeddings.

Findings

Experimental evaluations were conducted using multiple perturbation strategies, each applied with varying strengths. The performance of CDAE was compared against both the original BERT embeddings and SimCSE. The results indicated that CDAE consistently preserved higher embedding similarity when exposed to perturbations. This improvement became more pronounced as the strength of the perturbations increased. The framework demonstrated effectiveness in enhancing representation stability while simultaneously preserving the semantic information encoded within the embeddings.

Why This Matters

The findings suggest that learning perturbation-invariant representations is a promising direction for improving the stability and robustness of sentence embeddings. This enhancement in embedding stability, particularly under semantic-preserving perturbations, can lead to more reliable language model applications.