Overview
SelectInfer is an optimization framework designed for efficient Large Language Model (LLM) inference on resource-constrained edge devices. The framework operates by selectively loading and computing neurons, aiming to reduce memory footprint and computational demands without compromising task performance.
Research Context
Large Language Models exhibit capabilities across Natural Language Processing (NLP) tasks. However, their high computational and memory requirements present deployment challenges for edge devices. Existing approaches, such as coarse-grained pruning or quantization, can lead to accuracy compromises or necessitate re-training and fine-tuning.
Approach
SelectInfer utilizes an offline LLM profiler to identify both task-specific and general-purpose neurons. This profiling stage informs two primary optimizations:
- Selective Loading: This optimization reduces memory footprint by loading only a subset of neurons identified as important during the offline profiling stage.
- Selective Computation: At runtime, this optimization dynamically computes only the most relevant neurons, further reducing computational demands.
Findings
Evaluation across multiple datasets indicated that SelectInfer achieved reductions in both memory footprint and computation. These reductions were observed while preserving task performance.
Why This Matters
The observed reductions in memory footprint and computation, coupled with preserved task performance, represent a practical step towards enabling LLM deployment on edge devices. This addresses the challenge of deploying models with high computational and memory demands on resource-constrained hardware.