The impact of Copilot on quality and security in open-source software
This study analyzes the impact of AI-assisted code development, using GitHub Copilot as a case study, on both the quality and security of open-source software. Given the widespread adoption of OSS in everyday business processes, it is crucial to understand how new AI tools influence these ecosystems.
Research question and methodology
The central research question is: "How does Artificial Intelligence code development influence the quality and security of open-source software?"
This question is particularly relevant given the large-scale use of OSS in a wide range of sectors. The research uses a quantitative methodology and analyzes metrics such as maintainability issues, reliability issues, technical debt, security issues, and security hotspots, taking into account the influence of lines of code (LOC).
Research design and techniques
The study uses regression analysis and SonarQube scans to measure changes in quality and security over a two-year period: one year before and one year after the introduction of Copilot. The GitHub repositories used are divided into two groups: a treatment group and a control group. The treatment group consists of programming languages such as Python and JavaScript, while the control group includes languages such as C, C#, and R.
Results: productivity gains versus quality and security considerations
The findings show a two-sided effect of AI code development. On the one hand, Copilot significantly increases productivity, as evidenced by a 25% increase in LOC. On the other hand, this productivity gain is accompanied by an increase in maintainability issues, reliability issues, and technical debt. When controlling for LOC, Copilot's direct impact on these quality metrics decreases. This suggests that a large part of the observed effects is caused by the growth of the codebase itself.
Security insights and implications
In the area of security, the results show a nuanced picture. There has been a significant increase in security hotspots, while no clear direct relationship with actual security issues has been found. Specifically, there has been a 17.6% increase in security hotspots, which decreases to 3.9% when controlled for LOC. These findings underscore that AI code development cannot be viewed separately from the expertise and choices of developers. Although no direct significant effect of Copilot on quality and security has been established, the research does show that the use of AI tools by developers has a clear influence on these metrics.
