Overview
SmartRAG is a framework designed for deploying large language models (LLMs) as personal assistants directly on mobile devices. This framework addresses the challenges of privacy, low latency, and offline availability inherent in on-device AI, while operating within the computational constraints of edge hardware. SmartRAG structures an intelligent assistant through the coordination of four distinct modules: Perception, Memory, Focus, and Thinking. A central component of SmartRAG is EvoNER, a continuously learning named-entity recognizer. This recognizer can expand its label inventory incrementally using teacher-distilled updates, allowing the system to assimilate new entity types without requiring retraining of the underlying LLM.
Knowledge extracted by the system is stored within MRGraph, a three-layer knowledge graph that preserves provenance. During query processing, this knowledge is retrieved via a hybrid pipeline that combines graph traversal, lexical matching, and dense semantic search. The on-device LLM is selectively engaged for specific, high-value semantic tasks, including labeling, planning, and answer synthesis. This selective invocation strategy aims to bound inference costs. The framework has been evaluated on four distinct QA benchmarks: TriviaQA, Natural Questions, HotpotQA, and MultiHopQA.
Research Context
Deploying large language models directly on mobile devices for personal assistant functionalities presents a tension between the computational demands of large models and the strict hardware budgets of edge devices. Conventional approaches, such as model compression, are argued to be insufficient on their own to resolve this challenge. Instead, the research suggests that on-device intelligence requires decomposition into complementary functional roles. The objective is to achieve privacy, low latency, and offline operability for LLM-based assistants on mobile platforms.
Approach
SmartRAG's approach is centered on a modular architecture comprising Perception, Memory, Focus, and Thinking modules operating in coordination. EvoNER, the continually learnable named-entity recognizer, expands its label inventory through teacher-distilled updates. This mechanism allows EvoNER to assimilate previously unencountered entity types without necessitating the retraining of the foundational LLM. The extracted knowledge is then structured and stored in MRGraph, a three-layer knowledge graph designed to maintain provenance. Retrieval of this knowledge during query time is facilitated by a hybrid pipeline integrating graph traversal, lexical matching, and dense semantic search operations.
The on-device LLM's role is specifically limited to high-value semantic operations, which include labeling, planning, and the synthesis of answers. This design decision aims to control and limit the computational costs associated with LLM inference. The system uses a quantized 1.7-billion-parameter backbone for its LLM component.
Findings
- SmartRAG, utilizing a quantized 1.7B-parameter backbone, achieved multi-hop reasoning performance competitive with models up to 18 times larger.
- The framework operates entirely on commodity smartphones.
- SmartRAG functions within practical memory and latency envelopes on mobile devices.
- Evaluation was conducted on four QA benchmarks: TriviaQA, Natural Questions, HotpotQA, and MultiHopQA.
Why This Matters
The ability of SmartRAG to deliver competitive multi-hop reasoning on commodity smartphones with a significantly smaller model (1.7B parameters versus models 18x larger) indicates a potential pathway for deploying complex LLM functionalities directly on edge devices. This capability is achieved while adhering to practical memory and latency constraints, suggesting a method for enabling advanced AI functionalities that prioritize user privacy, offline availability, and low latency on mobile platforms.