Overview
This research investigates model-free reinforcement learning specifically for co-safe linear temporal logic (sc-LTL) objectives within finite Markov Decision Processes (MDPs). The study addresses challenges associated with direct sample-based bootstrap methods, such as temporal difference (TD) or Q-learning, which can fail to converge to optimal policies for this problem class. These convergence issues stem from the noncontractive nature and nonuniqueness of solutions to the Bellman equation in this context. To overcome these limitations, a novel two-step model-free reinforcement learning method is introduced, designed to ensure convergence to an optimal policy.
Research Context
The problem of achieving co-safe LTL objectives in finite MDPs can be simplified to maximal reachability objectives through a standard product construction. Conventional model-free reinforcement learning approaches, including direct sample-based bootstrap methods like TD learning or Q-learning, encounter difficulties when applied to this specific problem. The core issue is that these methods may not converge to optimal policies. This non-convergence is attributed to two key properties of the underlying mathematical framework: the noncontractive nature of the problem and the nonuniqueness of solutions to the Bellman equation.
Approach
The developed method is a two-step model-free reinforcement learning approach. The first step involves utilizing a discounted surrogate. The primary purpose of this discounted surrogate is to identify a 'clamp set'. This clamp set is crucial for resolving the issue of nonuniqueness in the solutions to the Bellman equation, which is a significant factor in the non-convergence of other methods. After the clamp set is identified, the second step of the method is applied. This second step consists of two components: undiscounted policy evaluation and greedy policy improvement. The entire process is structured to provide guarantees of finding an optimal solution.
Findings
The research established theoretical guarantees for the proposed two-step method. Specifically, it was proven that the policy evaluation step exhibits almost-sure convergence. Furthermore, the policy iteration algorithm itself is guaranteed to achieve finite termination at an optimal policy. These theoretical findings indicate that the method can reliably identify an optimal policy for co-safe LTL objectives in finite MDPs, addressing the non-convergence issues observed with direct bootstrap methods. The theoretical results were supported and confirmed through numerical experiments. These experiments were conducted on a stochastic grid world, providing an empirical validation of the method's performance and its ability to converge to an optimal policy in a practical setting.
Key Limitations Mentioned by Researchers
- The study focuses on finite Markov Decision Processes.
- The problem is reduced to maximal reachability objectives via standard product construction.
- The method specifically addresses issues arising from the noncontractive nature and nonuniqueness of solutions to the Bellman equation in the context of co-safe LTL objectives.