Overview
This research investigates the energy consumption, performance, and output quality implications of various vLLM inference engine configurations when serving Large Language Models (LLMs). The study evaluates three specific vLLM configuration options: attention kernel type, prefix caching, and chunked prefill. The analysis considers 5 open-weight LLMs and 5 distinct inference tasks. The methodology involved $9,000$ runs and $93,600$ individual measurements, focusing on energy consumption, latency, and accuracy, examining both main and interaction effects.
Research Context
Large Language Models are influencing the development and maintenance of software. These models are typically deployed in production environments utilizing inference engines such as vLLM, which are designed for efficient serving of pre-trained and highly configurable models. Existing research has frequently concentrated on model architectures and hardware acceleration aspects. However, the influence of inference engine configuration on three critical metrics—energy consumption, performance, and output quality—has remained underexplored.
Approach
The study employed a controlled methodology to systematically assess the impact of selected vLLM configuration options. The configurations under scrutiny were:
- Attention kernel type
- Prefix caching
- Chunked prefill
All possible combinations of these configurations were evaluated. The investigation involved 5 distinct open-weight LLMs and 5 various inference tasks. The experimental setup generated a dataset of $9,000$ individual runs, from which $93,600$ specific measures were collected. The primary analytical dimensions were energy consumption, latency, and accuracy. The research design aimed to identify both main effects of individual configuration options and interaction effects between these options and the diverse tasks. The analytical scope included the behavior of these configurations under the default vLLM serving configuration and the specified workloads.
Findings
- The studied configuration options significantly impact energy consumption and performance. This impact is primarily driven by the attention kernel type and prefix caching.
- Chunked prefill demonstrated a limited effect on energy and performance under the default vLLM serving configuration and the evaluated workloads.
- The observed effects are highly dependent on both the specific model being used and the workload applied. There was no single configuration identified as universally optimal across all conditions.
- Model choice emerged as the dominant factor influencing global trade-offs between energy, performance, and accuracy. Configuration tuning provided localized improvements along the Pareto frontier.
- Inference options were unexpectedly found to affect model accuracy.