Currently in EWS when layout-tests run notices any test failure, we re-run the tests. If the re-run also notices any test failure, we run the tests on clean tree (after unapplying the patch and building webkit again). Due to flaky tests, it is possible that the first and second runs have different failures, and no common test failure. In that case, our analysis logic will mark the build as passed, irrespective of clean tree results, especially after the change in https://trac.webkit.org/changeset/253049/webkit It's inefficient and wasteful to run tests on clean tree in such case. We shouldn't skip clean-tree run in such cases. Also we should update the build summary with the list of flaky tests, which would make it easier for bot-watchers to notice flaky tests. Few examples where first and second layout-test runs had different failures: https://ews-build.webkit.org/#/builders/24/builds/9565 https://ews-build.webkit.org/#/builders/24/builds/9446 https://ews-build.webkit.org/#/builders/24/builds/9394 https://ews-build.webkit.org/#/builders/24/builds/9393 https://ews-build.webkit.org/#/builders/24/builds/9352 https://ews-build.webkit.org/#/builders/32/builds/1744 https://ews-build.webkit.org/#/builders/32/builds/1708 https://ews-build.webkit.org/#/builders/32/builds/1703 https://ews-build.webkit.org/#/builders/30/builds/1783 https://ews-build.webkit.org/#/builders/30/builds/1696 https://ews-build.webkit.org/#/builders/30/builds/1648 https://ews-build.webkit.org/#/builders/31/builds/1905 https://ews-build.webkit.org/#/builders/31/builds/1737 https://ews-build.webkit.org/#/builders/31/builds/1519
Created attachment 389393 [details] Patch
Created attachment 389409 [details] Patch
Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/45
> In that case, our analysis logic will mark the build as passed, irrespective of clean tree results This sounds terrible. I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch.
(In reply to Alexey Proskuryakov from comment #4) > I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch. Agree, will do that instead.