Overview
The Doppelganger benchmark has been introduced to measure the ability of audio representations to match synthetic sound effects with the specific real recordings from which they were generated. This benchmark pairs 10,420 real audio clips across 34 everyday sound events with their audio-conditioned synthetic counterparts, and includes a controlled 7-class corpus. The evaluation demonstrates that off-the-shelf audio encoders do not consistently match these cross-boundary sound effects. A direct training approach, utilizing synthetic-real pairs, exhibited generalization capabilities on unseen sound events.
Research Context
Audio-conditioned generators now produce synthetic sound effects from existing real recordings. Consequently, both real and synthetic versions of sound events increasingly coexist within sound libraries and the corpora used for training audio models. Prior to this work, no dedicated benchmark existed to measure whether an audio representation could effectively match a synthetic clip to its precise real recording source.
Approach
The core of the research involved the creation of the Doppelganger benchmark. This benchmark consists of two primary datasets:
- 10,420 real clips across 34 distinct everyday sound events, each paired with an audio-conditioned synthetic twin generated from it.
- A controlled 7-class corpus.
Various audio encoders were evaluated on this benchmark. Initial evaluations used off-the-shelf audio encoders. Subsequently, two main training strategies for improving matching performance were explored:
- Training encoders on sound-event labels, a standard approach for making embeddings boundary-agnostic.
- Training encoders directly on synthetic-real pairs, where an encoder learns to associate a synthetic twin with its specific real source clip.
A human annotation baseline was also established to provide a comparative measure. This involved 49 listeners performing matching trials. Additionally, the study investigated human perception of synthetic sounds, specifically examining how often synthetic twins were perceived as real recordings. A generator-specific detector was also tested for its ability to differentiate these audio-conditioned synthetic twins from real recordings.
Findings
- Off-the-shelf audio encoders do not consistently manage to match sound effects across the synthetic-real boundary.
- Training an embedding on sound-event labels, intended to ignore the synthetic-real boundary, was effective for familiar sounds but resulted in performance degradation below the untrained encoder for new sounds.
- Training an encoder directly on synthetic-real pairs—a clip and its own synthetic twin—demonstrated generalization. This approach enabled the encoder to match synthetic clips to their exact real sources on sound events held out of training approximately 80% of the time, an increase from 61% for untrained encoders (with a chance baseline of 0.03%).
- This pair-training method did not meaningfully improve category-level recognition on unseen sound events.
- The learned matching was specific to a single generator. It was robust to changes in that generator's settings but failed when a different generator was used. Performance also collapsed for text-only generators that were tested.
- A human annotation baseline, involving 49 listeners, performed above chance but below the models on the same matching trials.
- Human listeners identified synthetic twins as real approximately 29% of the time.
- A generator-specific detector was able to perfectly distinguish these audio-conditioned synthetic twins from real recordings.
Why This Matters
The inability of standard audio encoders to differentiate between synthetic and real sound effects, and to match them accurately, highlights a gap in current audio representation capabilities. The development of a benchmark and a successful training methodology for explicit synthetic-real pairing addresses this challenge directly. This has implications for corpus integrity, sound library management, and the development of future audio models that interact with both generated and authentic sound data.