Lunch seminar: Continuous Delivery

Title: Multi-factor Approach for Flaky Test Detection and Automated Root Cause Analysis

Speaker: Azeem Ahmad, Linköping University

Abstract:  Re-running test cases is the most popular and adopted approach to detect flaky test in large scale industries. Re-running is costly because it consumes a lot of computing power. Google uses 2-16% of its testing budget just to re-run flaky tests. Re-running is unreliable because it is hard to determine the number of re-runs to find discrepancy in output. We developed, implemented and evaluated multi-factor approach to detect flaky test without re-running. The contributed factors are (1) number of test smells in the test cases, (2) whether test case failed after executing on the corresponding change in production code, (3) test case history, and (4) test case size. The machine learning algorithm is implemented to find relationships between the contributed factors and the possibility of test flakiness. Once the build is failed, our tool can determine whether the failed test case is a real failure or flaky.