Overview
This research details the IHLC system developed for the LT-EDI 2026 Shared Task, focusing on gender-inclusive rewriting and counter-narrative generation. The system employs distinct methodologies for each task: Low-Rank Adaptation (LoRA) fine-tuning for rewriting and an inference-time activation steering technique for counter-narrative generation. The approaches aim to transform biased text into inclusive alternatives while striving to maintain semantic meaning and contextual coherence.
Research Context
Gender-inclusive language generation seeks to produce inclusive text from biased inputs. This study addresses this objective within the framework of two specific tasks: gender-inclusive rewriting and the generation of counter-narratives. The work was submitted as part of the LT-EDI 2026 Shared Task, an event focused on language technology for ethical and inclusive development.
Approach
Gender-Inclusive Rewriting
- The system utilizes parameter-efficient Low-Rank Adaptation (LoRA) fine-tuning for gender-inclusive rewriting.
- LoRA is a method designed to adapt large language models efficiently by introducing a small number of trainable parameters.
Counter-Narrative Generation
- For counter-narrative generation, the system implements a compute-efficient inference-time representation engineering approach.
- This approach involves deriving a principal steering direction from contrastive hidden-state activations. This derivation is performed using principal component analysis (PCA).
- The derived steering direction is then injected into the intermediate representations of the Gemma-3-4B-it model during inference.
- This injection mechanism is intended to steer the model's behavior towards inclusive responses without requiring modification of the model's original weights.
- The activation steering is combined with constrained prompting to produce counter-narratives that are described as polite and contextually appropriate.
Findings
- For the gender-inclusive rewriting task, the LoRA fine-tuning approach achieved an official score of 80.00%.
- For the counter-narrative generation task, the activation steering approach, combined with constrained prompting, achieved an official score of 78.12%.
- A manual analysis of the steering behavior identified several specific failure modes:
- Semantic drift: Changes in meaning beyond the intended inclusive transformation.
- Residual bias leakage: The persistence of unwanted biases in the generated output.
- Layer sensitivity: Variations in effectiveness or behavior depending on which intermediate layer the steering is applied to.
- Over-steering: Excessive application of the steering direction leading to unnatural or distorted output.
- Text degeneration: A decline in the quality or coherence of the generated text.
Why This Matters
The findings indicate both the practical potential and the current limitations of activation steering as a lightweight alternative to modifying model parameters for achieving controllable and socially aligned language generation. The described techniques offer computationally efficient methods for addressing bias in language models, which could be relevant for developing more inclusive AI applications.