if the ImageDiff tool fails, then the diff_image function (in ChromiumPort, Tools/Scripts/webkitpy/layout_tests/port/layoutchromium.py) returns "None" to indicate the test passed. It does print an error in this case, but it is too easy to overlook it. It is probably important to find a cleaner way to handle this. Some options (thanks to dpranke): - the ideal option is to modify NRWT to handle internal errors so that tests are marked as "internal error, could not run test" instead of passed or failed. - modify the code to cause a test failure and outputs a diff, then the user can see that the diff is actually correct (false negatives may be safer than false positives) - optionally, have NRWT output a diff, with a hardcoded .png that literally says there was an internal error in the image.