Revolutionizing Online System Operations with Bian Que: A Flexible Agentic Framework
Operating and maintaining large-scale online engine systems, such as those found in search, recommendation, and advertising platforms, traditionally demands considerable human effort. These demands extend across various critical functions, including release monitoring, alert response, and the often-complex process of root cause analysis. A new agentic framework, dubbed Bian Que, has been introduced to address these significant operational challenges, specifically targeting the orchestration capabilities that are crucial for the practical deployment of LLM-based agents in such scenarios.
The Challenge of Large-Scale Online System Operations
The inherent suitability of Large Language Model (LLM)-based agents for operational scenarios is recognized, yet a critical bottleneck has impeded their widespread practical deployment. This bottleneck does not primarily lie in the reasoning capabilities of LLMs, but rather in their orchestration capability. Specifically, the challenge involves the precise selection of relevant data and applicable knowledge tailored to each individual operational event. This includes identifying pertinent metrics, logs, and change events, as well as applying handbook-defined rules and empirically derived practitioner experience.
The difficulties arise from two primary issues. First, indiscriminately feeding all available signals into the system can lead to dilution and hallucination, compromising the accuracy and effectiveness of the LLM agent. Second, manually curating the mapping between specific events and the necessary data and knowledge is a task that becomes intractable under the pressure of dozens of daily releases, a common occurrence in large online systems. The Bian Que framework aims to overcome these limitations by providing a structured and automated approach to this critical orchestration challenge.
Introducing Bian Que: An Agentic Operating Framework
Bian Que is presented as an agentic operating framework designed to streamline and enhance the operation and maintenance (O&M) of online engine systems. The framework incorporates three distinct contributions that collectively aim to improve efficiency, accuracy, and adaptability in handling operational events. These contributions are centered around a unified operational paradigm, a flexible skill arrangement mechanism, and a unified self-evolving mechanism.
Unified Operational Paradigm for Routine O&M Actions
One of the core contributions of the Bian Que framework is its unified operational paradigm. This paradigm abstracts routine daily O&M actions into three canonical patterns. This abstraction simplifies the complex interplay of various tasks that operators typically perform, providing a structured approach for LLM agents to understand and execute these operations. The three canonical patterns identified within this paradigm are:
- Release Interception: This pattern involves proactively monitoring and intervening during system releases to prevent or mitigate potential issues.
- Proactive Inspection: This entails continuous monitoring and analysis of system health and performance to identify potential problems before they escalate into critical events.
- Alert Root Cause Analysis: This focuses on systematically investigating and determining the underlying causes of system alerts, a crucial step in resolving issues and preventing recurrence.
By categorizing O&M actions into these well-defined patterns, Bian Que provides a clearer operational context for its agentic components, thereby improving the relevance and specificity of the data and knowledge applied during event handling.
Flexible Skill Arrangement for Context-Specific Operations
A second crucial contribution of the Bian Que framework is its flexible Skill Arrangement. This mechanism is central to addressing the orchestration bottleneck by ensuring that each specific operational context receives the precise data and knowledge it requires. Within this framework, each predefined 'Skill' explicitly delineates the requisite data and operational knowledge needed for a particular context. This explicit definition is key to avoiding the pitfalls of indiscriminate information feeding.
The adaptability of these Skills is a notable feature. They can be automatically generated and updated by LLM agents, leveraging the inherent capabilities of these models to learn and adapt. Furthermore, the framework allows for iterative optimization of these Skills by on-call engineers. This optimization is achieved through natural language instructions, enabling human operators to fine-tune and improve the system's operational capabilities based on their experience and evolving system needs. This hybrid approach, combining automated generation with human oversight via natural language, ensures both efficiency and high-quality operational responses.
Unified Self-Evolving Mechanism for Continuous Improvement
The third significant contribution of Bian Que is its unified self-evolving mechanism. This mechanism is designed to enable continuous improvement and adaptation of the framework over time. Each correction signal received by the system triggers two parallel evolutionary pathways, fostering a dynamic learning environment that enhances the agent's effectiveness.
- Distilling Event Memory into Knowledge: This pathway involves processing past operational events and their outcomes to extract valuable insights. These insights are then distilled into refined knowledge, enriching the system's understanding of various scenarios and improving its future decision-making capabilities.
- Targeted Refinement of Skills: The second pathway focuses on precisely refining the existing Skills based on correction signals. This targeted refinement ensures that the Skills remain relevant and effective as system behaviors evolve and new operational challenges emerge. It allows the framework to adapt its operational responses with greater precision and accuracy.
This self-evolving mechanism underpins Bian Que's ability to learn from experience, akin to how human operators gain expertise over time, thereby enhancing its long-term performance and robustness in dynamic online system environments.
Deployment and Performance on a Real-World System
The efficacy of the Bian Que framework has been demonstrated through its deployment on a real-world e-commerce search engine, specifically that of KuaiShou. The results from this deployment highlight significant improvements across several key operational metrics, indicating the practical benefits of the framework.
Reduced Alert Volume
One of the most notable achievements reported from the deployment is a substantial reduction in alert volume. Bian Que successfully reduced the volume of alerts by 75%. This reduction is critical for operational teams, as it minimizes alert fatigue and allows engineers to focus on more significant and complex issues, improving overall efficiency and response times.
High Root-Cause Analysis Accuracy
Root-cause analysis is a complex and often time-consuming task in system operations. Bian Que achieved an impressive 80% root-cause analysis accuracy. This high level of accuracy ensures that underlying problems are correctly identified, leading to more effective and lasting solutions, rather than addressing only the symptoms.
Cut in Mean Time to Resolution (MTTR)
The framework also delivered a significant improvement in the mean time to resolution (MTTR), cutting it by over 50%. A reduction in MTTR directly translates to less downtime and faster recovery from incidents, which is crucial for maintaining service availability and user satisfaction in online systems.
Offline Evaluation Pass Rate
In addition to its live operational performance, Bian Que demonstrated strong capabilities in offline evaluations. It attained a 99.0% pass rate on these evaluations, providing further validation of its reliability and precision in handling a diverse range of operational scenarios under controlled conditions. This high pass rate suggests a robust and well-engineered system capable of performing consistently.
Broader Implications for System Operation and Maintenance
The development and successful deployment of Bian Que indicate a significant step forward in automating and enhancing the operation and maintenance of large-scale online systems. By addressing the orchestration capabilities of LLM-based agents, the framework paves the way for more efficient and accurate operational responses, directly tackling the challenges posed by the increasing complexity and scale of modern online infrastructures.
The ability to precisely select relevant data and applicable knowledge, combined with a self-evolving mechanism, suggests a future where operational systems can continuously learn and adapt without constant manual intervention. This could lead to a paradigm shift in how companies manage their critical online services, moving towards more autonomous and intelligent operational frameworks.
The framework's contributions – the unified operational paradigm, flexible Skill Arrangement, and unified self-evolving mechanism – provide a comprehensive solution that mitigates common issues such as dilution and hallucination, which arise when LLMs are indiscriminately fed too much information. The empirical results from KuaiShou further underscore the practical value and effectiveness of Bian Que, offering a blueprint for similar challenges elsewhere in the industry. For further details, the codebase is available at https://github.com/benchen4395/BianQue_Assistant.