Bug 209912

Summary: [ews] Display list of test failures along-with the layout-test build step
Product: WebKit Reporter: Aakash Jain <aakash_jain>
Component: Tools / TestsAssignee: Aakash Jain <aakash_jain>
Status: RESOLVED FIXED    
Severity: Normal CC: aakash_jain, ap, jbedard, webkit-bug-importer, wilander
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=203411
Attachments:
Description Flags
Patch
none
Screenshot none

Aakash Jain
Reported 2020-04-02 09:41:52 PDT
Display list of test failures along-with the layout-test build step in the buildbot page. It would make it easy to notice the test failures without opening the complete 'layout test results' in a new tab. It would also make it easier to compare test-failures from the first run, re-run and re-run-without-patch.
Attachments
Patch (3.43 KB, patch)
2020-04-02 09:51 PDT, Aakash Jain 		no flags
DetailsFormatted DiffDiff
Screenshot (184.50 KB, image/png)
2020-04-02 14:35 PDT, Aakash Jain 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Aakash Jain
Comment 1 2020-04-02 09:51:00 PDT
Created attachment 395273 [details] Patch
Aakash Jain
Comment 3 2020-04-02 14:35:50 PDT
Created attachment 395302 [details] Screenshot
John Wilander
Comment 4 2020-04-02 14:39:19 PDT
Nice! Would it be possible to get the type of failure in that list too? Like text diff, timeout, crash, and such?
Jonathan Bedard
Comment 5 2020-04-02 14:56:24 PDT
(In reply to John Wilander from comment #4) > Nice! Would it be possible to get the type of failure in that list too? Like > text diff, timeout, crash, and such? Just curious, are engineers not generally aware of the stdout output: https://ews-build.webkit-uat.org/#/builders/34/builds/119/steps/10/logs/stdio? That has what you're looking for.
John Wilander
Comment 6 2020-04-02 15:02:35 PDT
(In reply to Jonathan Bedard from comment #5) > (In reply to John Wilander from comment #4) > > Nice! Would it be possible to get the type of failure in that list too? Like > > text diff, timeout, crash, and such? > > Just curious, are engineers not generally aware of the stdout output: > https://ews-build.webkit-uat.org/#/builders/34/builds/119/steps/10/logs/ > stdio? That has what you're looking for. That I don't know. I'm just generally in favor of minimal amounts of clicks, scrolling, and irrelevant information. If a patch failed, I want to see why immediately and I want to see as little other information as possible. The sooner I know something failed, the better. Personally, I would also love a "Stop processing" feature so that when I see some issue and I realize I need to fix my patch, I can immediately free up resources for others. Aakash and I have discussed that. It's for another change but in the same vein. Another thing we discussed was the ability to give a hint to EWS on which part of the test suite to test first, again striving for early failure. The quicker I know my patch is not ready for review or landing, the better for everyone.
Aakash Jain
Comment 7 2020-04-03 07:13:40 PDT
(In reply to Jonathan Bedard from comment #5) > Just curious, are engineers not generally aware of the stdout output I personally find it confusing since it also list many tests which flaked (which I am not interested in).
Aakash Jain
Comment 8 2020-04-03 07:14:33 PDT
(In reply to John Wilander from comment #4) > Nice! Would it be possible to get the type of failure in that list too? Like text diff, timeout, crash, and such? It's possible, but not straight-forward. I will attempt that in a separate patch.
EWS
Comment 9 2020-04-03 07:17:20 PDT
Committed r259457: <https://trac.webkit.org/changeset/259457> All reviewed patches have been landed. Closing bug and clearing flags on attachment 395273 [details].
Radar WebKit Bug Importer
Comment 10 2020-04-03 07:18:17 PDT
<rdar://problem/61258395>
Aakash Jain
Comment 11 2020-04-04 10:59:13 PDT
Deployed on the server. Seems to be working fine. e.g.: https://ews-build.webkit.org/#/builders/32/builds/6728
Note You need to log in before you can comment on or make changes to this bug.