ICANEWS

Engineering & Technology

DiscoTrace Reveals Rhetorical Strategy Discrepancies in Human and LLM Information-Seeking Q&A

A new research development, detailed in the paper arXiv:2604.15140v1 , introduces DiscoTrace, a novel method designed to identify and analyze the rhetorical str

DiscoTrace: Uncovering Answering Strategies in Information-Seeking Questions for Humans and LLMs

A new research development, detailed in the paper arXiv:2604.15140v1, introduces DiscoTrace, a novel method designed to identify and analyze the rhetorical strategies employed by both human answerers and Large Language Models (LLMs) when responding to information-seeking questions. This research provides a structured approach to understanding how different entities construct answers, offering insights into the underlying preferences and systematic behaviors in question answering (QA).

Introduction to DiscoTrace

The core of this research is DiscoTrace, a method specifically developed for representing and comparing answering strategies. DiscoTrace conceptualizes an answer not merely as a collection of facts, but as a structured sequence of rhetorical choices. This method goes beyond surface-level content analysis to delve into the strategic construction of responses.

We introduce DiscoTrace, a method to identify the rhetorical strategies that answerers use when responding to information-seeking questions.

The method operates by representing answers in a particular way: as a sequence of question-related discourse acts. These discourse acts are then paired with interpretations of the original question, forming a detailed analytical framework. A key component of this representation involves annotations built upon rhetorical structure theory (RST) parses. RST provides a framework for analyzing text organization, focusing on the relationships between different parts of a text. By integrating RST parses, DiscoTrace provides a granular view of how answer components are related and how they contribute to the overall rhetorical effect.

This detailed representation allows for a systematic comparison of answering strategies across different sources. The ability to identify and categorize these strategies is crucial for understanding the diversity, or lack thereof, in how information is delivered in response to queries.

Research Goal: Understanding Answering Strategies

The primary research goal of the study is to identify the rhetorical strategies that answerers utilize when providing responses to information-seeking questions. This goal applies to both human answerers and LLMs. By identifying these strategies, the researchers aim to gain a deeper understanding of the processes and preferences involved in answer construction. The research seeks to characterize these strategies in a quantifiable and comparable manner, moving beyond subjective evaluations of answer quality.

The exact research question, as framed by the introduction of DiscoTrace, revolves around defining and then identifying these rhetorical strategies. The method itself, by representing answers as a sequence of question-related discourse acts paired with interpretations of the original question, annotated on top of rhetorical structure theory parses, directly serves this goal. This focus on rhetorical strategies, rather than just content accuracy, highlights a critical dimension of information exchange.

Key Findings: Diverse Human Preferences vs. LLM Uniformity

The application of DiscoTrace yielded several significant findings, particularly concerning the differences in answering strategies between human communities and LLMs.

Human Communities Exhibit Diverse Preferences for Answer Construction

One of the central findings revealed by the DiscoTrace analysis is the inherent diversity in how different human communities construct their answers. The study applied DiscoTrace to answers sourced from nine distinct human communities. The results indicated that these communities do not adhere to a single, universal answering strategy. Instead, they demonstrate varied preferences regarding the rhetorical elements they choose to emphasize and the way they structure their responses.

This diversity suggests that human communication in QA is highly contextual and influenced by community-specific norms, expectations, or implicit guidelines. The rhetorical strategies employed by one community might prioritize conciseness and directness, while another might favor comprehensive explanations or a more narrative approach. This observation underscores the complex and multifaceted nature of human information exchange.

LLMs Lack Rhetorical Diversity Even When Prompted

In stark contrast to the human communities, LLMs exhibited a notable lack of rhetorical diversity in their answers. This finding is particularly striking because it persisted even when the LLMs were specifically prompted to mimic the answering guidelines of various human communities. Despite explicit instructions designed to encourage varied rhetorical approaches, LLMs consistently failed to replicate the rich diversity observed in human-generated responses.

This suggests a fundamental difference in how LLMs generate text compared to human cognition. While LLMs can process and produce information, their current capabilities may not extend to faithfully adopting and enacting a wide range of rhetorical strategies based on contextual guidelines. The uniformity in their rhetorical output, even under specific prompting conditions, indicates a limitation in their ability to dynamically adapt their answering style to match diverse human preferences.

LLMs Systematically Opt for Breadth

Another critical finding from the DiscoTrace analysis is that LLMs systematically opt for breadth in their responses. This means that LLMs tend to address a wider range of interpretations of the original question than human answerers. While human answerers often make strategic choices to focus on specific aspects or interpretations of a question, LLMs show a consistent tendency to cover as many possible interpretations as they can identify.

This systematic breadth implies a different approach to information processing and delivery. Humans might implicitly filter or prioritize interpretations based on perceived relevance, user intent, or community norms. LLMs, however, appear to adopt a more exhaustive strategy, attempting to provide information related to all plausible interpretations of a query. This behavior, while seemingly comprehensive, might not always align with optimal human information-seeking behavior, where targeted and concise answers are often preferred.

Methodology: DiscoTrace's Analytical Framework

The methodology underpinning these findings is DiscoTrace itself. The method offers a structured way to analyze the rhetorical construction of answers. It represents answers as a sequence of distinct components:

  • Question-related discourse acts: These are specific communicative actions within an answer that relate directly to the question. Examples are not provided in the source, but the concept implies identifiable units of rhetorical function within the response.
  • Interpretations of the original question: This component acknowledges that a single question can have multiple valid interpretations. DiscoTrace tracks which of these interpretations are addressed by different parts of the answer.

These two components are annotated on top of rhetorical structure theory (RST) parses. RST is a linguistic theory that describes text coherence by identifying relationships between clauses or groups of clauses. By using RST parses, DiscoTrace is able to analyze the hierarchical and functional relationships between different parts of an answer, providing a deep understanding of its rhetorical organization. The combination of discourse acts, question interpretations, and RST parses creates a powerful framework for dissecting and comparing answering strategies. Without this detailed methodological approach, it would be challenging to objectively quantify and compare the nuanced differences in rhetorical choices that the study highlights.

Implications for LLM Development

The findings from the DiscoTrace research have direct implications for the future development of LLM answerers. The observed discrepancies between human and LLM answering strategies point to areas where current LLMs can be improved to better align with human expectations and pragmatic communication needs.

Our findings can guide the development of pragmatic LLM answerers that consider a range of strategies informed by context in QA.

Specifically, the lack of rhetorical diversity in LLMs suggests a need to develop mechanisms that allow these models to adopt more varied and context-aware answering styles. If LLMs are to serve effectively across diverse domains and user groups, they must be capable of generating answers that reflect the specific rhetorical preferences of those contexts. This involves moving beyond a one-size-fits-all approach to answer generation.

Furthermore, the systematic tendency of LLMs to opt for breadth implies that future development should focus on refining their ability to discern and prioritize question interpretations. While comprehensive answers can be valuable, overly broad responses might lead to information overload or reduce efficiency, especially in scenarios where users seek specific, targeted information. Developing more pragmatic LLM answerers would involve enabling them to consider a range of strategies that are genuinely informed by context in QA, much like humans do. This could involve teaching LLMs to understand the implicit biases, norms, and preferences of different communities or interaction styles, leading to more human-like and effective communication.

Looking Ahead: Towards Context-Aware LLMs

The research sets a clear direction for what's next in the field of LLM development: creating LLM answerers that are more pragmatic and adaptable. The emphasis is on building models that can "consider a range of strategies informed by context in QA." This means moving beyond models that primarily focus on semantic accuracy and completeness, towards models that also master the art of rhetorical effectiveness and contextual relevance.

Future research stemming from DiscoTrace could explore methods to explicitly train LLMs on diverse rhetorical patterns found in human communities. This might involve curating datasets that are not just content-rich but also rich in rhetorical annotations, allowing models to learn the nuances of different answering styles. Another avenue could be developing sophisticated prompting techniques or architectural modifications that enable LLMs to dynamically select and apply rhetorical strategies based on immediate contextual cues, such as the perceived intent of the questioner or the characteristics of the target audience.

The goal is to bridge the gap between the current capabilities of LLMs and the sophisticated, context-informed communication strategies employed by humans. By focusing on rhetorical diversity and strategic breadth (or selectivity) in answering, the research aims to pave the way for LLMs that are not just intelligent, but also pragmatically wise in their interactions.

Key Findings