ICANEWS

Engineering & Technology

FastE: Readout-Triggered Token Compression for LLM Embedding Inference

Research has identified depth-dependent prefix redundancy within final-readout Large Language Model (LLM) embedding models. This redundancy was observed across

Overview

Research has identified depth-dependent prefix redundancy within final-readout Large Language Model (LLM) embedding models. This redundancy was observed across representative backbones, specifically Qwen3-Embedding and Qwen3-VL-Embedding. The study indicates that the removal of prefix states causes substantially greater damage in shallow layers compared to deeper layers, suggesting that prefix states become increasingly compressible as they, along with readout states, propagate through the network.

To address this, the researchers developed FastE, a method designed for token compression. FastE operates as a training-free, plug-and-play system. It utilizes a shared fixed threshold on batch-mean readout-prefix alignment. This threshold functions as a lightweight online heuristic, determining when compression should occur. Furthermore, FastE ranks prefix states based on the attention scores they receive from the readout position; this ranking dictates which states are retained for subsequent layers.

Research Context

The investigation focuses on the internal mechanics of final-readout LLM embedding models, specifically examining the behavior of prefix states and their redundancy. The identification of depth-dependent redundancy implies a structural characteristic of how information is processed and consolidated within these models. The observation that removal of prefix states is more detrimental in shallow layers points to an evolving importance or informational density of these states as they progress through the model's architecture.

Approach

The FastE method implements a two-pronged strategy for token compression:

  1. Online Heuristic for Compression Triggering: FastE employs a shared fixed threshold that operates on the batch-mean readout-prefix alignment. This alignment serves as an indicator to decide when to initiate compression.
  2. Prefix State Selection: To determine which prefix states to retain, FastE ranks them according to the attention scores they receive from the readout position. States with higher attention scores from the readout position are prioritized for retention in subsequent layers.

The method is characterized as training-free and plug-and-play, indicating that it does not require additional model training and can be integrated into existing systems.

Findings

  • Depth-Dependent Prefix Redundancy: The study identified that prefix redundancy in final-readout LLM embedding models is depth-dependent. This pattern was consistent across Qwen3-Embedding and Qwen3-VL-Embedding backbones.
  • Compressibility Evolution: It was found that removing prefix states leads to significantly more detrimental effects in shallow layers than in deeper layers. This suggests that prefix states become progressively more compressible as the prefix and readout states advance through the network.
  • Computational Cost Reduction: FastE demonstrated a substantial reduction in computational costs. Specifically, on the NarrativeQA dataset using Qwen3-Embedding-0.6B, FastE reduced decoder-backbone FLOPs by 40.11%.
  • Quality Retention: Despite the FLOPs reduction, FastE maintained 99.53% of the Full Forward nDCG@10 performance on the NarrativeQA dataset with Qwen3-Embedding-0.6B.
  • Customizable Trade-off: The method provides a customizable quality-efficiency trade-off. This customization is achieved through adjusting the maximum removal ratio, without requiring retraining, across multiple evaluation scenarios. These scenarios included five text embedding benchmarks, two backbone scales, and three cross-modal retrieval tasks.

Why This Matters

The development of FastE offers practical value for several systems. Its ability to reduce computational costs while largely preserving quality is relevant for scalable embedding generation. This has implications for applications such as information retrieval, indexing systems, clustering algorithms, and multimodal representation systems.

Potential Applications

The researchers indicate that FastE offers practical value for scalable embedding generation in several domains:

  • Retrieval systems
  • Indexing systems
  • Clustering algorithms
  • Multimodal representation systems

Key Findings

  • Depth-dependent prefix redundancy identified in final-readout LLM embedding models (Qwen3-Embedding, Qwen3-VL-Embedding).
  • Prefix state removal is more damaging in shallow layers, indicating increased compressibility at greater depths.
  • FastE reduces decoder-backbone FLOPs by 40.11% on NarrativeQA with Qwen3-Embedding-0.6B.
  • FastE retains 99.53% of Full Forward nDCG@10 on NarrativeQA with Qwen3-Embedding-0.6B.
  • The quality-efficiency trade-off is customizable via maximum removal ratio without retraining across benchmarks and tasks.