Bug 66581
Summary: | new-run-webkit-tests returns false-positives for chromium port if ImageDiff fails | ||
---|---|---|---|
Product: | WebKit | Reporter: | Shawn Singh <shawnsingh> |
Component: | Tools / Tests | Assignee: | Dirk Pranke <dpranke> |
Status: | RESOLVED DUPLICATE | ||
Severity: | Normal | CC: | dpranke, eric, ojan, simon.fraser, tony |
Priority: | P2 | Keywords: | NRWT |
Version: | 528+ (Nightly build) | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Bug Depends on: | |||
Bug Blocks: | 64491 |
Shawn Singh
if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed. It does print an error in this case, but it is too easy to overlook it.
It is probably important to find a cleaner way to handle this. Some options (thanks to dpranke):
- the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed.
- modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives)
- optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.
Attachments | ||
---|---|---|
Add attachment proposed patch, testcase, etc. |
Eric Seidel (no email)
I wonder if this is a dupe of or at least related to bug 47240.
Eric Seidel (no email)
On further reading, no. :)
Dirk Pranke
whoops, this is now a dup of bug 92934, since I attached a patch there ...
*** This bug has been marked as a duplicate of bug 92934 ***