Overview
Research explored a reinforcement-learning (RL) approach for controlling the turbulence-zonal-flow transition in the modified Hasegawa-Wakatani system, a minimal model of electrostatic drift-wave turbulence. The objective was to achieve bidirectional control, specifically turbulence suppression and inverse zonal-break tasks, using RL agents coupled with a GPU-native solver.
Research Context
Control of plasma turbulence represents a long-standing challenge within magnetically confined fusion research. The study utilized the modified Hasegawa-Wakatani system as its plasma model. This model incorporated a weak zonal drag, which damped otherwise long-lived zonal structures, providing them a finite lifetime. This damping mechanism restored the drive-damping balance necessary for repeatable transitions within finite control episodes. Actuation within the system was applied via a spatially distributed Gaussian source field, subject to a time-weighted budget constraint on actuation costs.
Approach
The plasma model was integrated with reinforcement learning agents. Specifically, CNN-based soft actor-critic and twin-delayed deterministic policy-gradient agents were employed. This integration leveraged a GPU-native JAX solver, optimized for fast online training. The RL approach was applied to two distinct control tasks:
- Turbulence Suppression Task: The goal was to minimize turbulent flux given a specific actuation budget.
- Inverse Zonal-Break Task: The objective was to disrupt existing zonal structures and sustain a turbulent state.
For the inverse zonal-break task, a physics-informed warm-buffer initialization was utilized to facilitate discovery, as random exploration alone was found to struggle with locating the optimal manifold within the action space.
Findings
- In the turbulence-suppression task, the learned budget-aware schedule achieved the lowest time-integrated turbulent flux. This performance was observed for a given consumed budget and across all tested unseen initial conditions. The learned schedule outperformed both constant and linearly decreasing baseline schedules.
- For the inverse zonal-break task, the RL agent discovered an actuation pattern characterized by up-down antisymmetry. This specific pattern induced radial $E \times B$ convection. The $E \times B$ convection, in turn, disrupted the zonal structure and sustained the turbulent state.
- The study indicated that physics-informed warm-buffer initialization played a critical role in the discovery process for the inverse zonal-break task.
Why This Matters
These results demonstrate the applicability of reinforcement learning as a practical trajectory optimizer for nonlinear plasma dynamics, specifically in areas such as turbulence control. The findings also underscore the importance of integrating physics guidance into such reinforcement learning applications.