Overview
A new unified benchmark, TalkFa, has been developed to address the absence of a comprehensive evaluation standard for Farsi dialogue generation and understanding. Farsi is spoken by over 120 million individuals. TalkFa integrates three distinct datasets, each designed to evaluate specific aspects of dialogue systems: WIKI-FADIAL for knowledge-grounded generation, DAILYDIALOG-FA for dialogue act and emotion recognition, and PLAYDIAL-FA for sentiment analysis in theatrical dialogues.
The benchmark's construction involved large language models (LLMs) for initial data generation, followed by a multi-stage review and revision process conducted by native Farsi speakers to ensure human approval for all released dialogues. Experimental evaluations were conducted using six LLAMA and MISTRAL models to assess performance across generation and understanding tasks. Further validation included human evaluations, independent external validation, and comparisons with an LLM judge (GPT-4.1) to assess the accuracy of automatic metrics. Zero-shot evaluations with frontier LLMs were also performed to gauge the benchmark's challenge level.
Research Context
The development of TalkFa is motivated by the identified lack of a comprehensive benchmark specifically tailored for Farsi in the domain of dialogue generation and understanding. This deficiency impacts the ability to reliably evaluate and advance dialogue systems for a language spoken by a significant global population.
Approach
The TalkFa benchmark was constructed as a unified system, comprising three distinct datasets:
- WIKI-FADIAL: This dataset contains 4,200 dialogues grounded in Wikipedia content, designed for evaluating knowledge-grounded generation capabilities.
- DAILYDIALOG-FA: Comprising 6,600 dialogues, this dataset includes annotations for dialogue acts and emotions, facilitating the assessment of dialogue understanding.
- PLAYDIAL-FA: This dataset features 2,100 theatrical dialogues, each annotated with sentiment labels, enabling the evaluation of sentiment recognition in a structured conversational context.
The data construction process involved leveraging LLMs to assist in generating initial dialogue content. Crucially, every dialogue underwent a multi-stage review and revision process by native Farsi speakers. Only dialogues that received final human approval were included in the released benchmark. This methodology aimed to ensure the quality and authenticity of the conversational data.
For experimental evaluation, six models from the LLAMA and MISTRAL families were utilized. The evaluation focused on both dialogue generation and classification tasks. The performance of LoRA (Low-Rank Adaptation) was specifically investigated for its impact on dialogue generation.
Validation of the benchmark included:
- Human evaluation.
- Independent external validation.
- Comparisons with GPT-4.1 acting as an LLM judge to contextualize automatic metric performance.
- Zero-shot evaluation with frontier LLMs to assess the benchmark's difficulty.
Findings
The experimental evaluation of TalkFa yielded several key findings across dialogue generation and understanding tasks:
- Dialogue Generation with LoRA: The application of LoRA significantly improved dialogue generation performance. This improvement was achieved while using only 25% to 50% of the training data, yet recovering over 90% of the final performance gains observed.
- Dialogue Act Performance: Among the models tested for classification tasks, FABERT achieved the best performance in dialogue-act recognition.
- Emotion Recognition Performance: LORA-MISTRAL-7B demonstrated the highest performance in emotion recognition.
- Sentiment Score: MISTRAL-24B achieved the highest sentiment score in the evaluations.
- Benchmark Reliability: Human evaluation and independent external validation confirmed the reliability of the TalkFa benchmark.
- Automatic vs. Human Metrics: Comparisons with GPT-4.1, utilized as an LLM judge, indicated that automatic metrics substantially overestimate the quality of dialogue generated.
- Benchmark Challenge: Zero-shot evaluation conducted with frontier LLMs suggested that TalkFa remains a challenging benchmark for current models.
Why This Matters
The creation of a comprehensive Farsi dialogue benchmark addresses a critical gap for researchers and developers working on conversational AI systems for Farsi speakers. This standardized evaluation tool provides a reliable means to assess, compare, and advance models in dialogue generation and understanding for a language spoken by over 120 million people.