Bug 66581

Summary: new-run-webkit-tests returns false-positives for chromium port if ImageDiff fails
Product: WebKit Reporter: Shawn Singh <shawnsingh>
Component: Tools / TestsAssignee: Dirk Pranke <dpranke>
Status: RESOLVED DUPLICATE    
Severity: Normal CC: dpranke, eric, ojan, simon.fraser, tony
Priority: P2 Keywords: NRWT
Version: 528+ (Nightly build)   
Hardware: Unspecified   
OS: Unspecified   
Bug Depends on:    
Bug Blocks: 64491    

Shawn Singh
Reported 2011-08-19 13:53:07 PDT
if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed. It does print an error in this case, but it is too easy to overlook it. It is probably important to find a cleaner way to handle this. Some options (thanks to dpranke): - the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed. - modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives) - optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.
Attachments
Eric Seidel (no email)
Comment 1 2011-10-11 13:15:56 PDT
I wonder if this is a dupe of or at least related to bug 47240.
Eric Seidel (no email)
Comment 2 2011-10-11 13:16:31 PDT
On further reading, no. :)
Dirk Pranke
Comment 3 2012-08-01 19:27:37 PDT
whoops, this is now a dup of bug 92934, since I attached a patch there ... *** This bug has been marked as a duplicate of bug 92934 ***
Note You need to log in before you can comment on or make changes to this bug.