Overview
This study investigates the speed at which development teams update their software dependencies, particularly focusing on vulnerable versions. It introduces two novel metrics, Mean-Time-To-Update (MTTU) and Mean-Time-To-Remediate (MTTR), designed to provide a more accurate reflection of update practices by addressing limitations in existing metrics. The research aims to assist practitioners in understanding how quickly packages update their dependencies, examining ecosystems such as npm, PyPI, and Cargo.
Research Context
Practitioners face increasing concerns regarding third-party dependencies with vulnerable versions, present both directly and transitively in their software. To mitigate this issue, projects are encouraged to promptly update to non-vulnerable dependency versions and to consider the update practices of the dependencies they integrate. Previous research has proposed metrics like Mean-Time-To-Update (MTTU) and Mean-Time-To-Remediate (MTTR) to quantify the responsiveness of package development teams in maintaining updated dependencies. The existing MTTU metric covers all dependencies, while MTTR specifically measures the time taken for a package to update its vulnerable dependencies. However, these established metrics reportedly fail to account for factors such as floating versions and the prioritization of recent updates. Such omissions can lead to inaccuracies in assessing a development team's actual update practices.
Approach
The study proposes two new metrics: an updated Mean-Time-To-Update for dependencies (MTTU) and a refined Mean-Time-To-Remediate for vulnerable dependencies (MTTR). These new metrics are designed to overcome the identified limitations of previous formulations, specifically addressing issues like floating versions and the emphasis on recent updates. An empirical study was conducted on a dataset comprising $163,207$ packages across three distinct ecosystems: npm ($117,129$ packages), PyPI ($42,777$ packages), and Cargo ($3,301$ packages). The research characterized the differences in MTTU and MTTR across these ecosystems. Furthermore, the study investigated various package characteristics that may influence these update times. A secondary objective was to determine the utility of MTTU as a proxy for MTTR in situations where sufficient vulnerability data is absent.
Findings
- Most packages examined demonstrated a relatively fast practice in updating their dependencies.
- The study characterized how the npm, PyPI, and Cargo ecosystems differed in terms of their MTTU and MTTR values.
- Various package characteristics were found to influence both MTTU and MTTR.
- When investigating whether MTTU could serve as a proxy for MTTR in the absence of adequate vulnerability data, the study did not find strong statistical evidence to support this.
- The findings suggest that MTTU could only be partially used as a proxy for MTTR, with caution, when vulnerability data is unavailable.
Why This Matters
Understanding the speed at which development teams update their dependencies, particularly vulnerable ones, is crucial for improving software supply chain security. The proposed refined metrics provide practitioners with tools to more accurately assess the update behaviors of component packages, which can inform choices about dependency adoption and maintenance strategies. The cautionary finding regarding the use of MTTU as a proxy for MTTR underscores the continued importance of specific vulnerability data for comprehensive security assessments.