| Summary: | [ews] Display flaky test names in build summary when ReRunWebKitTests passes | ||||||
|---|---|---|---|---|---|---|---|
| Product: | WebKit | Reporter: | Aakash Jain <aakash_jain> | ||||
| Component: | Tools / Tests | Assignee: | Aakash Jain <aakash_jain> | ||||
| Status: | RESOLVED FIXED | ||||||
| Severity: | Normal | CC: | aakash_jain, ap, jbedard, webkit-bot-watchers-bugzilla, webkit-bug-importer | ||||
| Priority: | P2 | Keywords: | InRadar | ||||
| Version: | Other | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| See Also: | https://bugs.webkit.org/show_bug.cgi?id=207504 | ||||||
| Attachments: |
|
||||||
|
Description
Aakash Jain
2020-01-31 09:20:29 PST
Created attachment 389364 [details]
Patch
This change will help in easily identifying flaky tests from builds like these (in which layout-tests have some test failures but re-run-layout-tests passes): https://ews-build.webkit.org/#/builders/24/builds/9685 https://ews-build.webkit.org/#/builders/24/builds/9686 https://ews-build.webkit.org/#/builders/24/builds/9661 https://ews-build.webkit.org/#/builders/24/builds/9600 https://ews-build.webkit.org/#/builders/24/builds/9588 https://ews-build.webkit.org/#/builders/24/builds/9592 Comment on attachment 389364 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review > Tools/BuildSlaveSupport/ews-build/steps.py:1360 > + flaky_failures_string = ', '.join(flaky_failures) This looks good for now. But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) > Tools/BuildSlaveSupport/ews-build/steps.py:-1355 > - message = 'Passed layout tests' Why did we move this out of the if statement. Comment on attachment 389364 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=389364&action=review >> Tools/BuildSlaveSupport/ews-build/steps.py:1360 >> + flaky_failures_string = ', '.join(flaky_failures) > > This looks good for now. > > But, once we have this information, we could cross reference these tests by name against the results database and see if they've failed in the last week. If they haven't, that's a decent indication of a newly flakey test (perhaps caused by the patch) Agree. Another thing we need to do is to automatically file bugs for these flaky tests. >> Tools/BuildSlaveSupport/ews-build/steps.py:-1355 >> - message = 'Passed layout tests' > > Why did we move this out of the if statement. No strong reason. Moving it back. Committed r255513: <https://trac.webkit.org/changeset/255513> Deployed on production server. |