Bug 58858

reftests only apply to the image part of the test, not the text part.

there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do.

(In reply to comment #1) > reftests only apply to the image part of the test, not the text part. But still, the test should pass, right? I was running with -p. (In reply to comment #2) > there are (artificial) samples in webkitpy.layout_tests.port/test.py that you can play with via new-run-webkit-tests --platform test, if that's helpful. Of course, they don't generate real output, so it probably isn't for what you're trying to do. I'll take a look.

(In reply to comment #3) > (In reply to comment #1) > > reftests only apply to the image part of the test, not the text part. > > But still, the test should pass, right? I was running with -p. > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.

I'll take a look. There is no sample 'real' mismatch reftests in LayoutTest directory. So I might miss something. I'll try to submit a sample mismatch reftest, expecting to find a issue. (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #1) > > > reftests only apply to the image part of the test, not the text part. > > > > But still, the test should pass, right? I was running with -p. > > > > Oh, I see, I read that as you had made the text output the same. Sorry. Yes, I would expect such a test to fail.

Created attachment 90142 [details] add-sample-reftests

I posted a patch which adds two sample reftests to the fast/harness directory. It seems a mismatch-reftest works correctly on my environment. I cannot reproduce the wrong behavior. Hmm. It would be great if you could help finding an issue about mismatch reftests. (In reply to comment #6) > Created an attachment (id=90142) [details] > add-sample-reftests

Comment on attachment 90142 [details] add-sample-reftests When I aply this patch and run: build-webkit new-run-webkit-tests LayoutTests/fast/harness/ I get: Regressions: Unexpected image mismatch : (2) fast/harness/sample-fail-mismatch-reftest.html = IMAGE fast/harness/sample-mismatch-reftest.html = IMAGE The images are similar for first test and different for the second, but they both fail due to image mismatch. Note this is not a Chromium DRT, it's the regular mac build of WebKit. I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.

Hi Dmitry, Thank you for testing that. I can reproduce that on mac build of WebKit. I'll take a look further. Sorry for the inconvenience. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file. (In reply to comment #8) > (From update of attachment 90142 [details]) > When I aply this patch and run: > > build-webkit > new-run-webkit-tests LayoutTests/fast/harness/ > > I get: > > Regressions: Unexpected image mismatch : (2) > fast/harness/sample-fail-mismatch-reftest.html = IMAGE > fast/harness/sample-mismatch-reftest.html = IMAGE > > The images are similar for first test and different for the second, but they both fail due to image mismatch. > Note this is not a Chromium DRT, it's the regular mac build of WebKit. > > I would expect only one of those to fail, and also it would be nice to add the 'expected to fail' test to other platform's test_expectations.txt file.

Let me note what I figured out: - WebKitPort, which is used in new-run-webkit-tests, is trying to disable 'pixel_tests' in default. - We should skip reftests if pixel_tests is disabled. - Even if '--pixel-tests' in turned on explicitly on WebKitPort, there is a difference between ChromiumLinuxDriver and WebKitDriver about how to handle 'image_hash' value. Such a different behavior makes a correct reftests fail on Mac build of WebKit. I'll file bugs for each.

I filed the two bugs: - https://bugs.webkit.org/show_bug.cgi?id=59078 - https://bugs.webkit.org/show_bug.cgi?id=59188

I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. After the patch in bug 59188 is landed, I am going to add these sample reftests too.

Bug 59188 was landed. I'd like to commit these sample mismatch reftests to detect a regression. Should I rename the subject of this bugzilla entry or create a new bug for these sample tests? (In reply to comment #12) > I posted a patch to fix a WebKitDriver so that reftests work in bug 59188. > After the patch in bug 59188 is landed, I am going to add these sample reftests too.

It might be better to close this bug. A mismatch refrests should work on all platfoms after fixing bug 59188. I'll file a separate bug for adding sample mismatch reftests.