Comparing module- and class-level regression test selection in continuous integration

Zhao, Peiyuan

Permalink

https://hdl.handle.net/2142/101059 Copy

Description

Title

Comparing module- and class-level regression test selection in continuous integration

Author(s)

Zhao, Peiyuan

Issue Date

2018-04-24

Director of Research (if dissertation) or Advisor (if thesis)

Marinov, Darko

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Software Testing
• Regression Testing

Abstract

Developers rely on regression testing in their continuous integration (CI) environment to find if their changes introduce faults. While regression testing is widely practiced, it is very costly. Regression test selection (RTS) reduces the cost of regression testing by running only the tests that are affected by the changes. Industry has adopted module-level RTS running in a CI environment. Researchers have recently proposed class-level RTS but did not evaluate it in a CI environment. In this work, we compare module-level and class-level RTS techniques in a CI environment. Specifically, we evaluate GIB, a module-level RTS technique, and Ekstazi, a class-level RTS technique, on Travis CI, a popular CI service for open-source projects. We also propose a hybrid RTS technique, GIBstazi, that combines aspects of the module-level and class-level RTS techniques. GIBstazi is safer than Ekstazi because GIBstazi tracks non-source-code changes while Ekstazi does not. We evaluate all the techniques on Travis CI using real, historical commits that were built on Travis CI. Our results show that all the RTS techniques do save testing time compared to running all tests (RetestAll) in a CI environment, but the savings in the end-to-end time for a full build is not as high as shown in previous work in a local environment running with a dedicated machine due to the extra overhead in a CI environment. Moreover, we are the first to inspect test failures from RetestAll builds. We find that these failures are often not selected by the RTS techniques, but almost all test failures missed by the RTS techniques are from so called flaky tests, which can pass or fail nondeterministically regardless of the code changes. The overall results show that RTS can be beneficial for the developers, hybrid GIBstazi gives the best trade-off, and RTS not only saves time but also avoids misleading developers by flaky test failures.

Graduation Semester

2018-05

Type of Resource

text

Permalink

http://hdl.handle.net/2142/101059

Copyright and License Information

Copyright 2018 Peiyuan Zhao

Owning Collections