Bug 93434

Summary: [NRWT] REGRESSION(r124967): New tests without expected results handled as failures
Product: WebKit Reporter: Csaba Osztrogonác <ossy>
Component: Tools / TestsAssignee: Csaba Osztrogonác <ossy>
Status: RESOLVED FIXED    
Severity: Normal CC: abarth, dpranke, ojan, ossy, rniwa, webkit.review.bot
Priority: P2 Keywords: NRWT
Version: 528+ (Nightly build)   
Hardware: All   
OS: All   
Bug Depends on:    
Bug Blocks: 88702    
Attachments:
Description Flags
Patch none

Csaba Osztrogonác
Reported 2012-08-07 22:17:27 PDT
Before r124967 new tests without expected files were handled as new tests, not as failures. Which means only orange buildbot, not red buildbot. But after r124967, new tests make the bot red, which is not the expected behaviour. Let's see the NRWT's output: before r124967: ---------------- Unexpected flakiness: text diff mismatch (4) fast/forms/range/slider-mouse-events.html = TEXT PASS fast/forms/range/slider-onchange-event.html = TEXT PASS http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS http/tests/security/cross-frame-access-put.html = TEXT PASS Regressions: Unexpected no expected results found : (2) fast/table/table-row-outline-paint.html = MISSING svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING and the buildbot reports: 4 flakes 2 missing results after r124967: --------------- Unexpected flakiness: text failures (4) fast/forms/range/slider-mouse-events.html = TEXT PASS fast/forms/range/slider-onchange-event.html = TEXT PASS http/tests/security/contentSecurityPolicy/object-src-url-allowed.html = TEXT PASS http/tests/security/cross-frame-access-put.html = TEXT PASS Regressions: Unexpected missing results : (2) fast/table/table-row-outline-paint.html = MISSING svg/as-image/animated-svg-repaints-completely-in-hidpi.html = MISSING and the buildbot reports: 2 failures 4 flakes It seems it is a similar bug as https://bugs.webkit.org/show_bug.cgi?id=93346
Attachments
Patch (1.79 KB, patch)
2012-08-07 22:29 PDT, Csaba Osztrogonác 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Csaba Osztrogonác
Comment 1 2012-08-07 22:20:11 PDT
Yes, it is so similar bug ... The new "Regressions: Unexpected missing results" match for the regexp of failures .. Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg ----------------------------------------------------------- ... def _parseNewRunWebKitTestsOutput(self, logText): incorrectLayoutLines = [] expressions = [ ('flakes', re.compile(r'Unexpected flakiness.+:?\s*\((\d+)\)')), ('new passes', re.compile(r'Expected to .+, but passed:\s+\((\d+)\)')), ('missing results', re.compile(r'no expected results found\s*:\s+\((\d+)\)')), ('failures', re.compile(r'Regressions: Unexpected.+:?\s*\((\d+)\)')), ] testFailures = {} ...
Csaba Osztrogonác
Comment 2 2012-08-07 22:29:05 PDT
Created attachment 157115 [details] Patch
Csaba Osztrogonác
Comment 3 2012-08-07 22:31:46 PDT
If we really want to change the output of NRWT, please make it sync with master.cfg changes. And please contact with Lucas and/or Bill to restart the master after the change to ensure correct behaviour. And please ping me, because I have to make similar changes in master.cfg of build.webkit.sed.hu. (Maybe you have to make similar changes in master.cfg of build.chromium.org too)
Csaba Osztrogonác
Comment 4 2012-08-07 22:35:00 PDT
Comment on attachment 157115 [details] Patch Clearing flags on attachment: 157115 Committed r124988: <http://trac.webkit.org/changeset/124988>
Csaba Osztrogonác
Comment 5 2012-08-07 22:35:07 PDT
All reviewed patches have been landed. Closing bug.
Dirk Pranke
Comment 6 2012-08-07 22:40:49 PDT
Comment on attachment 157115 [details] Patch Sorry! (again).
Dirk Pranke
Comment 7 2012-08-07 22:42:20 PDT
(In reply to comment #6) > (From update of attachment 157115 [details]) > Sorry! (again). To be clear(er), it was not actually my intention to change the output that the bot would be seeing :(. I did not mean to break the parser (or require changes to the parser).
Csaba Osztrogonác
Comment 8 2012-08-07 23:27:16 PDT
not problem ;) I didn't notice that the order of two messages changed (printed for each test, printed at the end), so my fix wasn't correct. I fixed it - https://trac.webkit.org/changeset/124994
Note You need to log in before you can comment on or make changes to this bug.