Overview
This research introduces FinProBench, a benchmark designed for evaluating artificial intelligence (AI) agents in professional financial tasks. Central to FinProBench is the Role-Grounded Rubric Construction (RGRC) methodology, a pipeline for developing evaluation rubrics based on deliverables produced by human practitioners in specific professional roles. The benchmark and methodology aim to address the challenge of evaluating financial AI agents against real-world professional standards, which often encompass tacit knowledge not fully captured by criteria derived solely from task prompts or model outputs.
Research Context
The evaluation of AI agents, particularly in specialized domains like finance, requires robust criteria that align with actual professional work expectations. Existing rubric generation methods frequently derive criteria from task prompts or model outputs. However, such approaches may overlook implicit standards and quality indicators evident in deliverables produced by human professionals. This gap suggests a need for evaluation frameworks that directly incorporate the characteristics and standards embedded within real-world professional output.
Approach
The study developed the RGRC pipeline, which comprises four stages: Deliverable Collection, Competency Extraction, Rubric Synthesis, and Validation. This pipeline is designed to derive rubrics from practitioner deliverables within the same professional role, capturing tacit standards and enabling the distinction of various quality levels. The rubrics generated through RGRC are also intended to be transferable across different tasks within a given role.
Prior to analysis, the researchers classified 57 occupations based on their deliverable genres. This classification resulted in two categories: 30 'prior-rich conventional roles' and 27 'prior-sparse role-specialized roles'. FinProBench itself was constructed from 1,723 curated deliverables, covering 57 occupations, 8 financial sub-industries, and 161 deliverable types. An initial evaluation set of 20 complete tasks, covering 20 roles in 7 sub-industries, was released as part of the benchmark.
The evaluation used heterogeneous Large Language Model (LLM) judges and role-level rubrics. The performance of human deliverables was compared against four systems, with all demonstrating overlapping 95% confidence intervals.
Findings
- The RGRC methodology demonstrated distinct performance characteristics depending on the nature of the professional role.
- For 'conventional roles', prompt-only rubric generation nearly matched RGRC, with RGRC achieving 90.7% compared to prompt-only's 89.2%.
- For 'role-specialized roles', RGRC substantially outperformed prompt-only methods, achieving 99.1% compared to prompt-only's 78.0%. This indicates that while prompt engineering can approximate rubrics when conventions are well-represented in model priors, professional grounding becomes essential for standards beyond those priors.
- Human deliverables, on average, ranked first in evaluations using heterogeneous LLM judges and role-level rubrics, scoring 73.7 out of 100. The four systems evaluated scored 70.3, 70.2, and 69.6.
- Reusing rubrics at the role level was estimated to reduce per-task construction effort by 6.7 times compared to authoring each rubric from scratch.
Why This Matters
The RGRC pipeline offers a method for developing evaluation rubrics that incorporate tacit professional standards directly from practitioner deliverables, addressing a limitation of existing rubric generation techniques. The observed performance disparity between conventional and role-specialized roles highlights the importance of grounding evaluation criteria in professional practice, particularly for complex or less standardized tasks. The potential for significant reduction in rubric construction effort through role-level reuse could streamline the development of evaluation benchmarks for specialized AI agents.