Bug 248544

Summary:	REGRESSION (256665@main): [ Ventura ] fast/images/avif-as-image.html is a consistent image failure
Product:	WebKit	Reporter:	Karl Rackler <rackler>
Component:	New Bugs	Assignee:	Said Abou-Hallawa <sabouhallawa>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	ryanhaddad, sabouhallawa, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=247704 https://bugs.webkit.org/show_bug.cgi?id=250045

Description Karl Rackler 2022-11-30 11:52:40 PST

Description:
fast/images/avif-as-image.html

There was an AVIF tests garden at 255947@main where I first saw the history of the image failure for Ventura.  I observed later in the history at 255984@main that AVIF image decoding using libavif and dav1d is enabled, and that is where I see the test begin to pass on Monterey and BigSur, but fail for Ventura. Still, there was a test expectation in place for this.  Finally, there was work to AVIF image animates only one loop at 256665@main where the test expectation was removed, which seemed to cause the image failure on the flakiness dashboard to appear again.  

History:
https://results.webkit.org/?suite=layout-tests&test=fast%2Fimages%2Favif-as-image.html&platform=ios&platform=mac&limit=50000

Image Diff:
https://build.webkit.org/results/Apple-Ventura-Release-WK2-Tests/257176@main%20(385)/fast/images/avif-as-image-diffs.html

Comment 1 Radar WebKit Bug Importer 2022-11-30 11:53:06 PST

<rdar://problem/102823196>

Comment 2 Karl Rackler 2022-11-30 13:24:59 PST

This issue can be reproduced using the command: 
run-webkit-tests --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --iterations 50 -f --no-retry fast/images/avif-as-image.html

Result: 
Regressions: Unexpected image-only failures (1)
  fast/images/avif-as-image.html [ ImageOnlyFailure ]

Comment 3 Karl Rackler 2022-11-30 13:28:10 PST

I have marked this test as a consistent image failure while this issue is investigated.

Comment 4 EWS 2022-11-30 13:38:47 PST

Test gardening commit 257195@main (60a1d6da7b93): <https://commits.webkit.org/257195@main>

Reviewed commits have been landed. Closing PR #6985 and removing active labels.

Comment 5 Said Abou-Hallawa 2022-12-07 14:33:50 PST

Pull request: https://github.com/WebKit/WebKit/pull/7287

Comment 6 EWS 2022-12-12 10:44:03 PST

Committed 257738@main (c7d2aef57843): <https://commits.webkit.org/257738@main>

Reviewed commits have been landed. Closing PR #7287 and removing active labels.

Comment 7 Karl Rackler 2023-01-06 10:13:19 PST

Re-opening.  Looking at the history, landing 257738@main did not seem to resolve the issue.

Comment 8 Said Abou-Hallawa 2023-01-06 17:25:17 PST

The TestExpectations have these entries for this test:

LayoutTests/TestExpectations:fast/images/avif-as-image.html [ ImageOnlyFailure ]
LayoutTests/platform/win/TestExpectations:fast/images/avif-as-image.html [ Skip ]
LayoutTests/platform/mac/TestExpectations:[ Ventura+ ] fast/images/avif-as-image.html [ Pass ]

What I wanted to say by these entries is: this test is expected to fail on all platforms and on all macOS versions except on macOS Ventura.

So I am not sure why iOS says it fails even though the expectations say: "yes we expect it to fail on iOS". Also I do not know why it runs on macOS Monetary and BigSur arm64. Should I add entries in mac-ventura/TestExpectations,  mac-monterey/TestExpectations and  mac-ventura/TestExpectations for this file?

Comment 9 Karl Rackler 2023-01-06 18:08:49 PST

(In reply to Said Abou-Hallawa from comment #8)
> The TestExpectations have these entries for this test:
> 
> LayoutTests/TestExpectations:fast/images/avif-as-image.html [
> ImageOnlyFailure ]
> LayoutTests/platform/win/TestExpectations:fast/images/avif-as-image.html [
> Skip ]
> LayoutTests/platform/mac/TestExpectations:[ Ventura+ ]
> fast/images/avif-as-image.html [ Pass ]
> 
> What I wanted to say by these entries is: this test is expected to fail on
> all platforms and on all macOS versions except on macOS Ventura.
> 
> So I am not sure why iOS says it fails even though the expectations say:
> "yes we expect it to fail on iOS". Also I do not know why it runs on macOS
> Monetary and BigSur arm64. Should I add entries in
> mac-ventura/TestExpectations,  mac-monterey/TestExpectations and 
> mac-ventura/TestExpectations for this file?

You currently have the test expectations set up such that everything is passing on WebKit Results Database if you turn on "Filter expected results" except for mac-ventura.

This is because the test expectation for Mac-ventura was removed by 257738@main.

I am looking at the latest run, and the expected image is a lime square, and the actual image is a black square.  
https://build.webkit.org/results/Apple-Ventura-Release-AppleSilicon-WK2-Tests/258574@main%20(917)/fast/images/avif-as-image-diffs.html

I'm adding the test expectation back in place while this issue is investigated.

Comment 10 EWS 2023-01-06 18:16:15 PST

Test gardening commit 258590@main (eda93cc0bc7f): <https://commits.webkit.org/258590@main>

Reviewed commits have been landed. Closing PR #8342 and removing active labels.

Comment 11 Said Abou-Hallawa 2023-01-13 10:58:08 PST

Pull request: https://github.com/WebKit/WebKit/pull/8630

Comment 12 EWS 2023-01-15 11:13:24 PST

Committed 258934@main (9bffbcdd799e): <https://commits.webkit.org/258934@main>

Reviewed commits have been landed. Closing PR #8630 and removing active labels.

Comment 13 Said Abou-Hallawa 2023-01-17 12:05:19 PST

*** Bug 250045 has been marked as a duplicate of this bug. ***