It is an irritating fact of Web Development that tests are flaky sometimes. Many customers deal with this by re-running tests one or more times when they fail in any given build.
There is currently no way to tell from the Sauce UI whether or not a test was re-run. You can guess by checking whether a test with a given name and platform was run more then once in a given build, and check if one failed and was followed by one which passed, but this is frustrating. In addition, re-runs change the number of tests in a build making it harder to compare what should be "Identical" builds.
It would be great if Sauce Labs annotated re-ran tests somehow.
Possible Annotation methods:
- Allow customers to use the REST API to mark a test as a re-ran test, including the Sauce ID of the original test
- Allow customers to include a desired capability to have Sauce Labs guess if a test is re-ran; Do this by comparing platform and name, as well as start and finish times, and pass/fail statuses. Only do this for tests in the same build.
Possible Annotation outcome:
- Re-ran tests show the number of re-runs on the Test Details page
- Re-ran tests include a tab linking to other instances of that test on another tab
- "Collapsing" re-ran tests into a single Test Details page
- Filters in the archives to show re-ran tests
- Counters on the Build tab to show re-ran tests
- Excluding all but one test result in the Build Details page, to keep numbers consistent (potentially allowing customers to choose whether to use the last result, or any successful result)
- The ability to find tests that re-run often via Analytics (to help addressing flaky tests)