Proactively detecting unreliable tests

Gyori, Alex

Permalink

https://hdl.handle.net/2142/98348 Copy

Description

Title

Proactively detecting unreliable tests

Author(s)

Gyori, Alex

Issue Date

2017-07-10

Director of Research (if dissertation) or Advisor (if thesis)

Marinov, Darko

Doctoral Committee Chair(s)

Marinov, Darko

Committee Member(s)

• Parthasarathy, Madhusudan
• Xie, Tao
• Lahiri, Shuvendu K

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Unreliable tests
• Flaky tests
• Proactive detection

Abstract

Regression testing is the most wide-spread method to ensure the quality of software systems. Whenever a change is made to the software, tests are run to ensure bugs are not introduced: if all tests pass, the change is merged into the codebase; otherwise, the developer needs to identify the bug that was introduced by the change. Developers assume that the outcomes of the tests in the regression testing process are reliable, i.e., that the failure indicates a bug introduced by the change. Unfortunately, unreliable tests manage to get into the test suite, slowing down the developers’ workflow and having developers debug not their software but rather the test code or infrastructure. This dissertation presents two techniques to enable developers to more easily and proactively detect and debug unreliable tests early, before they become a problem and slow down the development process. Developers write unreliable tests, which may pass on their machine but may at a later point fail because the environment changes. This dissertation presents a technique to detect when code makes wrong assumptions on underdetermined APIs. While these assumptions may hold in the current environment, they may not hold in the future, causing tests to fail. The technique, NonDex, detects such wrong assumptions by exploring different behaviors that may not manifest in the current implementation, but are allowed by the specification. Furthermore, the dissertation presents PolDet, a technique to detect when tests pollute their environment; such polluting tests can cause other tests to fail or pass seemingly nondeterministically because of the environment pollution rather than changes in the code. The two techniques enable developers to identify when they are introducing unreliability in their test suite and help identify the root causes of the unreliable tests. The results of the evaluation on open-source projects show the techniques are effective at identifying issues in open-source code and also that developers are eager to fix the issues and adopt the tools.

Graduation Semester

2017-08

Type of Resource

text

Permalink

http://hdl.handle.net/2142/98348

Copyright and License Information

Copyright 2017 Alex Gyori

Owning Collections