Overview
This study presents a commit-open protocol for detecting silent model substitution in hosted Large Language Models (LLMs). The protocol addresses a perceived incentive for hosted-LLM providers to advertise a stronger model while serving users with a computationally less expensive substitute. It is designed to close a specific vulnerability inherent in probe-after-return schemes, such as SVIP, where a dishonest provider could potentially route verifier probes to the advertised model while directing ordinary users to a substitute.
Research Context
Hosted LLMs face a challenge where providers might be incentivized to perform "silent substitution," meaning they serve a cheaper model while advertising a more capable one. Existing probe-after-return verification methods, exemplified by SVIP, introduce a "parallel-serve side-channel." This side-channel arises because a provider could theoretically distinguish between verification probes and user requests, directing probes to the advertised, stronger model and user requests to a substitute model. The research specifically aims to develop a mechanism that prevents this type of differential serving behavior during auditing.
Approach
The proposed commit-open protocol operates in two main phases. Prior to an "opening request," the LLM provider commits to a per-position sparse-autoencoder (SAE) feature-trace sketch of the served output. This commitment is made via a Merkle tree at a published probe layer. A verifier then opens (reveals) random positions within this committed trace. These opened positions are scored against a publicly available named-circuit probe library, which is calibrated to account for cross-backend noise. A decision on whether substitution has occurred is made using a fixed-threshold joint-consistency z-score rule.
The protocol's instantiation involved three distinct LLM backbones: Qwen3-1.7B, Gemma-2-2B, and a 4.5x scaled-up version, Gemma-2-9B, utilizing a 131k-feature SAE. The evaluation involved a set of 17 different attacker types. These included:
- Same-family lifts (substitute models from the same model family but weaker).
- Cross-family substitutes (models from different families).
- Rank-128 adaptive LoRA (a specific type of low-rank adaptation attack).
Findings
- The commit-open protocol rejected all 17 tested attacker types, which included same-family lifts, cross-family substitutes, and rank-128 adaptive LoRA.
- Rejection of these attackers occurred at a shared, scale-stable threshold.
- The same 17 attacker types successfully evaded detection by a matched SVIP-style parallel-serve baseline. This suggests the commit-open protocol effectively closes the specific side-channel exploited by parallel-serve schemes.
- A white-box end-to-end attack, employing backpropagation through the frozen SAE encoder, did not overcome the detection margin of the protocol.
- The limits of a feature-forgery attacker, one that never runs $M_{hon}$, were formally bounded in closed form through an intrinsic-dimension argument.
- The commitment step of the protocol introduced an overhead of approximately 2.1% to the forward-only wall-clock time when processing a batch of 32 requests.
Why This Matters
This research offers a method to enhance the verifiability and trustworthiness of hosted LLM services. By addressing the potential for silent model substitution, it could contribute to greater transparency regarding the actual computing resources and model capabilities utilized by service providers versus what is advertised. This may foster more accountable practices in the deployment and operation of LLMs in hosted environments.