Bug 207267

Summary: [ Mac wk2 ] webgpu/whlsl/ensure-proper-variable-lifetime.html is flaky ImageOnlyFailure.
Product: WebKit Reporter: Jason Lawrence <Lawrence.j>
Component: WebGPUAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: rackler, tsavell, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Mac   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=207199
https://bugs.webkit.org/show_bug.cgi?id=207200
Attachments:
Description Flags
Update Test Expectations none

Description Jason Lawrence 2020-02-05 07:38:14 PST
webgpu/whlsl/ensure-proper-variable-lifetime.html

Description:
This test is flaky failing on Mac wk2. There are other webgpu tests with similar results which I will relate.

History:
https://results.webkit.org/?suite=layout-tests&test=webgpu%2Fwhlsl%2Fensure-proper-variable-lifetime.html&limit=50000

Diff:
https://build.webkit.org/results/Apple-Catalina-Debug-WK2-Tests/r255810%20(2131)/webgpu/whlsl/ensure-proper-variable-lifetime-diff.png
Comment 1 Radar WebKit Bug Importer 2020-02-05 07:38:37 PST
<rdar://problem/59187793>
Comment 2 Jason Lawrence 2020-02-05 07:47:58 PST
Created attachment 389806 [details]
Update Test Expectations
Comment 3 Truitt Savell 2020-02-05 07:54:11 PST
Comment on attachment 389806 [details]
Update Test Expectations

Clearing flags on attachment: 389806

Committed r255811: <https://trac.webkit.org/changeset/255811>
Comment 4 Jason Lawrence 2020-02-05 15:55:51 PST
I am able to reproduce this issue with the following command:
run-webkit-tests xxx webgpu/whlsl/ensure-proper-variable-lifetime.html --iterations 5000 -f --exit-after-n-failures 5 -g --force

[106/5000] webgpu/whlsl/ensure-proper-variable-lifetime.html failed unexpectedly (reference mismatch)
[212/5000] webgpu/whlsl/ensure-proper-variable-lifetime.html failed unexpectedly (reference mismatch)
[259/5000] webgpu/whlsl/ensure-proper-variable-lifetime.html failed unexpectedly (reference mismatch)
[290/5000] webgpu/whlsl/ensure-proper-variable-lifetime.html failed unexpectedly (reference mismatch)
[617/5000] webgpu/whlsl/ensure-proper-variable-lifetime.html failed unexpectedly (reference mismatch)
Exiting early after 5 failures. 614 tests run.


609 tests ran as expected, 5 didn't (4386 didn't run):


Regressions: Unexpected image-only failures (1)
  webgpu/whlsl/ensure-proper-variable-lifetime.html [ ImageOnlyFailure ]
Comment 5 Jason Lawrence 2020-02-05 15:56:56 PST
I was able to reproduce the issue on 255856.
Comment 6 Karl Rackler 2020-09-03 17:59:47 PDT
Description:
webgpu/whlsl/ensure-proper-variable-lifetime.html

This is a flaky imageonlyfailure.

The first image failure that I see this on the dashboard was 10/152019 at r251120, but it was not a commit that seemed relevant that would have caused this.

It appears the original test expectations were set to failure instead of imageonlyfailure.  Modifying/correcting text expectation in https://trac.webkit.org/changeset/266564/webkit

I attempted to reproduce the imageonlyfailure using the previously used reproduction steps, but was unable to reproduce using those or other combinations I tried at this time.

The test was expecting a blue square outline with white background and center, and received a white square outline with black background and center.
History:
https://results.webkit.org/?suite=layout-tests&test=webgpu%2Fwhlsl%2Fensure-proper-variable-lifetime.html&limit=50000

Diff:
https://build.webkit.org/results/Apple-Catalina-Debug-WK2-Tests/r266535%20(6417)/webgpu/whlsl/ensure-proper-variable-lifetime-diff.png