Bug 207064

Summary: [ews] Do not run tests without patch when there are no consistent failures in two layout-test runs
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED INVALID    
Severity: Normal CC: aakash_jain, ap, jbedard, webkit-bot-watchers-bugzilla
Priority: P2    
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Aakash Jain 2020-01-31 12:30:02 PST 
Currently in EWS when layout-tests run notices any test failure, we re-run the tests. If the re-run also notices any test failure, we run the tests on clean tree (after unapplying the patch and building webkit again).

Due to flaky tests, it is possible that the first and second runs have different failures, and no common test failure. In that case, our analysis logic will mark the build as passed, irrespective of clean tree results, especially after the change in https://trac.webkit.org/changeset/253049/webkit

It's inefficient and wasteful to run tests on clean tree in such case. We shouldn't skip clean-tree run in such cases. Also we should update the build summary with the list of flaky tests, which would make it easier for bot-watchers to notice flaky tests.

Few examples where first and second layout-test runs had different failures:

https://ews-build.webkit.org/#/builders/24/builds/9565
https://ews-build.webkit.org/#/builders/24/builds/9446
https://ews-build.webkit.org/#/builders/24/builds/9394
https://ews-build.webkit.org/#/builders/24/builds/9393
https://ews-build.webkit.org/#/builders/24/builds/9352

https://ews-build.webkit.org/#/builders/32/builds/1744
https://ews-build.webkit.org/#/builders/32/builds/1708
https://ews-build.webkit.org/#/builders/32/builds/1703


https://ews-build.webkit.org/#/builders/30/builds/1783
https://ews-build.webkit.org/#/builders/30/builds/1696
https://ews-build.webkit.org/#/builders/30/builds/1648

https://ews-build.webkit.org/#/builders/31/builds/1905
https://ews-build.webkit.org/#/builders/31/builds/1737
https://ews-build.webkit.org/#/builders/31/builds/1519
Comment 1 Aakash Jain 2020-01-31 12:56:20 PST 
Created attachment 389393 [details]
Patch
Comment 2 Aakash Jain 2020-01-31 13:53:11 PST 
Created attachment 389409 [details]
Patch
Comment 3 Aakash Jain 2020-01-31 13:54:31 PST 
Sample run: https://ews-build.webkit-uat.org/#/builders/16/builds/45
Comment 4 Alexey Proskuryakov 2020-01-31 14:01:39 PST 
> In that case, our analysis logic will mark the build as passed, irrespective of clean tree results

This sounds terrible.

I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch.
Comment 5 Aakash Jain 2020-01-31 14:20:29 PST 
(In reply to Alexey Proskuryakov from comment #4)
> I think that we do want to re-run in this situation though. E.g. if try and retry both have 5 different failures, and retry without patch has 0 failures, that sounds like a failure for the patch.
Agree, will do that instead.