Overview
TRACE (Tool Retrieval via Augmented Chain-of-thought and Enterprise rules) is a two-stage curriculum designed to address limitations in parametric tool retrieval for Enterprise Large Language Models (LLMs). This curriculum aims to preserve parametric tool knowledge during training while enabling efficient, real-time deployment through single-beam greedy decoding.
Research Context
Parametric retrieval methods enable LLMs to retrieve tools by assigning each API a unique virtual token, which the model is then trained to generate via constrained beam search. Previous work, specifically Toolsense, identified two critical drawbacks in this regime: the destruction of parametric tool knowledge during training and the computational inefficiency of beam-search decoding for real-time deployment. TRACE was developed to resolve this dissociation.
Approach
The TRACE curriculum is structured into two stages:
Stage 1: Multi-format Memorization SFT
This initial stage reuses the multi-format memorization Supervised Fine-Tuning (SFT) approach from ToolSense. The objective of this stage is to seed tool knowledge within the LLM. It employs LoRA (Low-Rank Adaptation) for fine-tuning.
Stage 2: Tool Retrieval via Augmented Chain-of-thought and Enterprise Rules
This constitutes the core contribution of TRACE. In this stage, the model is trained to emit a thinking trace prior to generating a JSON list of tool tokens. The training data for Stage 2 is derived from two sources:
- RRB (Retrieval, Reasoning, and Boundary) pairs, also sourced from ToolSense.
- Queries synthesized to specifically target business rules curated by domain experts.
Both data sources are augmented with reasoning traces. This specific training objective in Stage 2 is designed to maintain the accuracy achieved in Stage 1 for MCQ (Multiple Choice Question) and QA (Question Answering) probing, while simultaneously enabling single-beam greedy decoding at production latency.
Findings
- Tool Understanding Preservation and Improvement: TRACE training for Stage 2 not only preserves but also improves tool understanding. Specifically, MCQ accuracy showed gains of +3.2 percentage points (pp) and QA probing accuracy gained +9 pp compared to Stage 1 performance.
- Retrieval Performance: Evaluated on a combined enterprise catalog comprising over 8,300 tools across two enterprise product lines, TRACE achieved approximately 86% recall on Domain A and approximately 60% recall on Domain B.
- Comparison to Baselines: These retrieval performances compare to embedding baseline performance of approximately 27% for Domain A and approximately 52% for Domain B.
- Deployment Capability: The system operates with single-beam greedy decoding, enabling direct deployment at production latency.
Why This Matters
The TRACE curriculum addresses identified challenges in implementing parametric tool retrieval for enterprise LLMs by preserving essential tool knowledge during training and ensuring the system operates efficiently enough for real-time applications. The observed improvements in tool understanding and retrieval recall, alongside production-latency decoding, make it suitable for operational deployment in enterprise environments.