The Impact of Copilot on Quality and Security in Open-Source Software
This research explores the impact of artificial intelligence code development, specifically focusing on GitHub Copilot, on the quality and security of open-source software (OSS). With OSS becoming a crucial part of software ecosystems and the rise of data breaches and hacks, this study investigates the influence of AI code development on the quality and security of OSS.
The research question addressed is ”How does Artificial Intelligence code development impact the quality and security of open source software?” This topic is particularly important due to the large adoption of OSS in day to day operations. The study employs a quantitative research methodology, the research explores metrics including maintainability issues, reliability issues, technical debt, security issues, and security hotspots, while accounting for the influence of lines of code (LOC).
The study employs regression analysis and Sonarqube scans to determine the quality and security fluctuation over a period of 2 years. One year before the release and one year after. GitHub repositories used in the study are split in two groups, one group the treatment group and the other the control group. The treatment group entails programming languages such as Python and JavaScript and the control group contains C, C#, and R. The findings reveal a dual-edged nature of AI code development. Firstly, Copilot significantly enhances productivity, as evidenced by a 25% increase in LOC. In contrast, this increase in productivity is followed by rises in maintainability issues, reliability issues, and technical debt. When controlling for LOC, the direct impact of Copilot on these quality metrics decreases, suggesting that much of the observed effects are mediated by the increase in code base. Security metrics reveal mixed results, with a significant increase in security hotspots but no clear association with security issues, revealing the nuanced role of AI in software development. This is indicated by a 17.6% increase in security hotspots, which reduces to a small 3.9% increase after controlling for LOC. These results show a shift in how AI code development should be approached, highlighting the interplay between AI code development and developer expertise. The results did not find a direct significant impact of Copilot on quality and security but did demonstrate the influence of developer usage of AI code development on these metrics.
