Overview
Research introduces Geometric Evolution Maps (GEMs) as a method for extracting stable concept probes from the residual streams of transformer architectures. This approach addresses the rotational dynamics of concept directions during their formation within these models. A GEM records the directional trajectory of a concept across a specific Concept Allocation Zone (CAZ) segment, identifying its settled direction at that segment's terminal layer. Subsequently, it locates the handoff layer, defined as the first layer immediately following the segment boundary where the settled direction can be evaluated outside the window used for its estimation. The methodology emphasizes that the handoff layer is derived from the segment boundary rather than being detected through a rotation criterion. A concept is typically understood to occupy multiple segments, and the cumulative set of these GEMs for a given concept constitutes its 'atlas'. This atlas, rather than any individual GEM, is presented as the output unit for subsequent applications. GEMs provide the ablation-targeting methodology utilized by the CAZ Validation paper within the same research program.
Research Context
Traditional methods for concept probing often involve selecting a fixed late layer or a point corresponding to the peak of a separation curve. However, this study identifies a structural characteristic of concept formation within transformer residual streams: the probe direction undergoes substantial rotation during the assembly phase. This rotation persists until after the concept has formed within its Concept Allocation Zone (CAZ). Ignoring this dynamic rotation can compromise the reliability of concept probes. The observed rotation is described as substantial and consistent across diverse architectures and concept types, indicating it is an inherent property of the representation itself, rather than an artifact of the estimation process.
Approach
The GEM methodology is structured around several key components:
- Directional Trajectory Recording: A GEM tracks the evolving direction of a concept across a specific CAZ segment.
- Settled Direction Identification: At the final layer of a CAZ segment, the method determines the concept's settled direction.
- Handoff Layer Location: The handoff layer is identified as the immediate layer beyond the segment boundary. This layer is the first point where the concept's settled direction, estimated within the preceding segment, can be evaluated independently of that estimation window. The determination of this handoff layer is based on the segment boundary, not on a dynamic rotation criterion.
- Concept Atlas Creation: Since a single concept typically spans multiple CAZ segments, its complete set of GEMs across these segments forms its 'atlas'. This atlas is designated as the comprehensive unit delivered for downstream utilization.
Findings
The research yielded several findings regarding concept evolution and probing in transformer models:
- Substantial and Consistent Rotation: Concept probe directions rotate significantly during assembly within transformer residual streams. This rotation persists until after the concept's formation within its Concept Allocation Zone (CAZ). This phenomenon is consistent across different architectures and concept types, indicating it is an intrinsic property of the representation.
- Causal Extraction: The direction extracted using GEMs is described as causal. Ablating this extracted direction suppresses substantially more separation compared to ablating a randomly chosen direction. However, the effect is not carried by a single site exclusively.
- Improved Probing Precision: Probing concepts after the completion of this directional rotation, as facilitated by GEMs, yields greater precision than probing during the rotational phase.
- Depth Advantage: A depth-matched control experiment indicated that the precision advantage of probing after rotation completion stems from probing at greater depth within the model, rather than any specific privilege associated with the handoff boundary itself.
- Structured Exceptions: A small number of structured exceptions to these observations were documented, rather than being left unaccounted for.
Why This Matters
The study provides a structured methodology for identifying and extracting stable concept probes from transformer models, which is crucial for understanding internal model representations. By accounting for the dynamic rotation of concept directions, GEMs offer a more precise and causally grounded approach to probing. This improved precision supports methodologies that require targeted ablation for validation, such as those used in the CAZ Validation paper within the same research program.