Overview
Research introduces an information-flow perspective for the specification and verification of explainability requirements within multi-agent systems. This approach frames explainability as a positive flow of information, enabling agents interacting with a system to acquire knowledge regarding the causes of observed effects. The methodology concurrently addresses the balance between this positive information flow and potential negative information flow, which could, for instance, compromise privacy guarantees. The core of this framework leverages epistemic temporal logic, augmented with quantification over counterfactual causes. This logical extension is applied to define explainability as a system-level requirement, specifically ensuring that a multi-agent system provides sufficient information for agents to gain knowledge about why certain effects occur. An algorithm has been developed to check finite-state models against these specifications, and its functionality has been demonstrated through a prototype implementation evaluated on several benchmarks.
Research Context
Explainable systems are characterized by their capacity to expose information about the underlying reasons for observed effects to the agents interacting with them. This exposure is conceptualized as a positive information flow. Concurrently, privacy considerations often involve restricting certain information flows, constituting what is termed negative information flow. Both the concepts of explainability and privacy fundamentally require rigorous reasoning about knowledge. The research addresses this need by employing formal methods capable of handling epistemic reasoning.
Approach
The proposed approach utilizes epistemic temporal logic as its foundational framework. This logic is extended specifically with quantification over counterfactual causes. This extension enables the formal specification that a multi-agent system exposes adequate information, thereby allowing interacting agents to acquire knowledge concerning the causal factors of a particular effect. This principle underpins the formulation of explainability as a system-level requirement. The methodology further includes an algorithm designed for checking finite-state models against these defined specifications. This algorithm facilitates the verification process, allowing for the assessment of whether a system adheres to its stipulated explainability and privacy requirements. A prototype implementation of this algorithm has been developed to demonstrate its practical application and effectiveness.
Findings
- The research demonstrates a method to specify explainability as a system-level requirement using epistemic temporal logic extended with counterfactual causation. This specification ensures multi-agent systems expose information such that agents acquire knowledge about why specific effects occurred.
- An algorithm was developed for checking finite-state models against these explainability specifications. This algorithm provides a mechanism for formal verification.
- A prototype implementation of the algorithm was presented and evaluated using several benchmarks.
- The evaluation illustrated that the approach can distinguish between systems that are explainable and those that are unexplainable according to the defined specifications.
- The approach also demonstrated the capability to incorporate and verify additional privacy requirements alongside explainability.
Why This Matters
The framework allows for the specification and verification of explainability as a system-level requirement. It also provides a method to balance this positive information flow for explainability against negative information flow needed for privacy. The development of an algorithm and prototype implementation for finite-state model checking offers a practical tool for system designers to assess adherence to these dual requirements.