Unstable / flaky tests are tests which sometimes pass and sometimes fail, without any code change. Obviously, they slow down developments when
developers have to rerun failed tests. However, the real problem is that you can’t completely trust these tests, they might fail for many different
reasons and you don’t know if any of them will happen in production.
Some tools, such as TestNG, enable developers to automatically retry flaky tests. This might be acceptable as a temporary solution, but it should
eventually be fixed. The more flaky tests you add, the more chances there are for a bug to arrive in production.
This rule raises an issue when the annotation org.testng.annotations.Test is given a successPercentage argument with a
value lower than 100.
Noncompliant code example
import org.testng.annotations.Test;
public class PercentageTest {
@Test(successPercentage = 80, invocationCount = 10) // Noncompliant. The test is allowed to fail 2 times.
public void flakyTest() {
}
}
