Overview
The research introduces unix-ctf, a procedural generator specifically designed to create capture-the-flag (CTF) tasks for shell agents. This system aims to create a training surface that focuses on reinforcing Unix competence, distinguishing it from general programming ability via a terminal. Each task generated by unix-ctf involves hiding a unique token (a flag) within a fresh Linux container using a single Unix feature, which the agent must then recover.
Research Context
Current terminal benchmarks often conflate Unix competence with general programming fluency. For instance, a solver proficient in Python but weak in Unix may still pass a substantial portion of Terminal-Bench 2.0. This conflation means that the specific skill profile of strong Unix competence with weaker programming is less frequently evaluated. The distinction between using shell and operating-system primitives as first-class toolsversus merely writing programs through a terminal is made operational in this work. The goal is to isolate and enable training for the Unix component of this skill set.
Approach
The unix-ctf system employs an LLM-assisted synthesis pipeline for task generation. This pipeline involves several stages:
- Candidate Hiding Technique Generation: An LLM generates potential methods for hiding flags.
- Script Rewriting: These techniques are then rewritten into parameterized hide-and-find script pairs.
- Filtering with Bidirectional Contract: The generated hide and find scripts are filtered based on a specific contract. The hide script must ensure no plaintext trace of the flag remains on disk, while the find script must successfully recover the flag within a fresh directory.
The LLM's role is restricted to generating the planting and recovery steps; the container, layout, and grading harness are fixed components of the pipeline. This constrained role for the LLM contributes to the pipeline's efficiency, landing 656 out of 750 raw attempts as portable, reusable variants, representing an 87.5% success rate. In contrast, a reproduction of Endless Terminals' full-container-generation approach achieved only a 17.4% success rate under the identical checking conditions. The 656 variants generated by unix-ctf canonicalize to 155 distinct techniques.
Findings
- The LLM-assisted synthesis pipeline achieved an 87.5% success rate in generating portable, reusable task variants (656 out of 750 raw attempts).
- These 656 variants condensed into 155 distinct Unix competence techniques.
- A reproduction of Endless Terminals' full-container-generation approach yielded only a 17.4% success rate under the same verification checks.
- Fine-tuning a Qwen3-8B model using LoRA with GRPO on the unix-ctf training surface increased its solve rate from 11.6% to 43.6% on a 15-skill multi-family holdout set ($n=225$).
- This fine-tuning also redistributed which InterCode-CTF tasks the model was able to solve and resulted in a +33 percentage point gain in Forensics tasks.
- The fine-tuned model achieved a score of 32/100 on InterCode-CTF.
Why This Matters
The results of this research suggest that Unix competence is a separable skill, capable of being trained independently, and is best evaluated directly rather than as a secondary aspect of programming through a shell. The development of unix-ctf provides a dedicated and efficient platform for reinforcing and measuring this specific skill set in AI agents.