Bug 244677

Summary:	New Test?: [ macOS wk1 ] media/media-source/media-source-istypesupported-vp8-without-vp9.html is a constant text failure
Product:	WebKit	Reporter:	Robert Jenner <jenner>
Component:	Media	Assignee:	Jean-Yves Avenard [:jya] <jean-yves.avenard>
Status:	RESOLVED FIXED
Severity:	Normal	CC:	eric.carlson, jean-yves.avenard, jer.noble, rackler, webkit-bot-watchers-bugzilla, webkit-bug-importer, youssefdevelops
Priority:	P2	Keywords:	InRadar
Version:	WebKit Nightly Build
Hardware:	Unspecified
OS:	Unspecified

Description Robert Jenner 2022-09-01 10:26:44 PDT

media/media-source/media-source-istypesupported-vp8-without-vp9.html

is a constant text failure on macOS wk1. 

HISTORY:
https://results.webkit.org/?suite=layout-tests&test=media%2Fmedia-source%2Fmedia-source-istypesupported-vp8-without-vp9.html

It is also slowing down EWS. Observed here:
https://ews-build.webkit.org/#/builders/73


TEXT DIFF:
 These tests may be expected to fail if the WebKit port does not support the format.
 
 Forcing VP9 decoder to be enabled.
-EXPECTED (MediaSource.isTypeSupported('video/webm; codecs=vp8') == 'true') OK
+EXPECTED (MediaSource.isTypeSupported('video/webm; codecs=vp8') == 'true'), OBSERVED 'false' FAIL
 Forcing VP9 decoder to be disabled.
-EXPECTED (MediaSource.isTypeSupported('video/webm; codecs=vp8.0') == 'true') OK
+EXPECTED (MediaSource.isTypeSupported('video/webm; codecs=vp8.0') == 'true'), OBSERVED 'false' FAIL
 END OF TEST

DIFF URL:
https://build.webkit.org/results/Apple-BigSur-Release-AppleSilicon-WK1-Tests/254041@main%20(12600)/media/media-source/media-source-istypesupported-vp8-without-vp9-pretty-diff.html

Comment 1 Radar WebKit Bug Importer 2022-09-01 10:27:03 PDT

<rdar://problem/99446320>

Comment 2 Karl Rackler 2022-09-01 11:14:53 PDT

I have marked this test as a flaky failure while this issue is investigated.

Comment 3 EWS 2022-09-01 11:19:08 PDT

Test gardening commit 254054@main (066fa32708f5): <https://commits.webkit.org/254054@main>

Reviewed commits have been landed. Closing PR #3921 and removing active labels.

Comment 4 Karl Rackler 2022-09-01 11:23:04 PDT

REPRODUCTION STEPS
I can reproduce this on 254044@main ToT.

Command: 
run-webkit-tests --debug -1 --exit-after-n-failures 1 --exit-after-n-crashes-or-timeouts 1 --iterations 50 -f --no-retry media/media-source/media-source-istypesupported-vp8-without-vp9.html

Result: 
Regressions: Unexpected text-only failures (1)
  media/media-source/media-source-istypesupported-vp8-without-vp9.html [ Failure ]

Comment 5 Ryan Haddad 2022-09-07 10:11:42 PDT

This test is failing on release as well, we need to remove the [ Debug ] flag from the expectation.

However, I don't know if webm is expected to work on WK1. Maybe this test needs to be skipped.

Comment 6 Jonathan Bedard 2022-09-08 07:41:59 PDT

Pull request: https://github.com/WebKit/WebKit/pull/3921

Comment 7 EWS 2022-09-08 07:53:13 PDT

Test gardening commit 254270@main (683811c96127): <https://commits.webkit.org/254270@main>

Reviewed commits have been landed. Closing PR #4128 and removing active labels.

Comment 8 Jean-Yves Avenard [:jya] 2022-12-13 03:36:01 PST

Pull request: https://github.com/WebKit/WebKit/pull/7543

Comment 9 Jean-Yves Avenard [:jya] 2022-12-13 16:18:35 PST

VP8 and VP9 are not supported in WK1, so the test is expected to fail. Amend the expectations to expect failure.

So the failure is to be expected

Comment 10 EWS 2022-12-18 22:29:49 PST

Committed 258071@main (96015181185f): <https://commits.webkit.org/258071@main>

Reviewed commits have been landed. Closing PR #7543 and removing active labels.

Comment 11 Robert Jenner 2023-01-10 15:40:38 PST

As stated above the failure is expected for WK1 platforms. So marking the expectation as Failure is the correct fix. Closing.