Bug 221218

Summary: [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html is flakey failing
Product: WebKit Reporter: Robert Jenner <jenner>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: RESOLVED WORKSFORME    
Severity: Normal CC: ben_schwartz, jer.noble, tsavell, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch
none
Patch
none
Update TestExpectations none

Description Robert Jenner 2021-02-01 11:12:49 PST 
imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html

Is flakey failing on MacOS Big Sur Wk2.

HISTORY URL:
https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html

DIFF URL:
https://build.webkit.org/results/Apple-BigSur-Release-WK2-Tests/r272143%20(1311)/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-diff.txt

DIFF TEXT:
--- /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-expected.txt
+++ /Volumes/Data/slave/bigsur-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize-actual.txt
@@ -1,3 +1,5 @@
 
-PASS Tests webm video-only frame size changes.
+Harness Error (TIMEOUT), message = null
 
+TIMEOUT Tests webm video-only frame size changes. Test timed out


Bug Reproducible on r272141, same failure verified.

Bug Produced using test: 
run-webkit-test imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html  --iterations 1000 -f

Regression point appears r272140, where the test did not crash. 

Bug appears to have been introduced in r272141
Comment 1 Radar WebKit Bug Importer 2021-02-01 11:13:15 PST 
<rdar://problem/73837300>
Comment 2 Robert Jenner 2021-02-01 14:23:27 PST 
We are unable to find a true regression point. This test appears to be very flakey. 

Tested back to r271039 just to see, and it did fail.
Comment 3 Robert Jenner 2021-02-11 12:18:54 PST 
Created attachment 420026 [details]
Patch
Comment 4 Ryan Haddad 2021-02-11 12:53:19 PST 
(In reply to Robert Jenner from comment #2)
> We are unable to find a true regression point. This test appears to be very
> flakey. 
> 
> Tested back to r271039 just to see, and it did fail.
I think this test has just been flaky since it was unskipped as part of https://trac.webkit.org/changeset/270720/webkit
Comment 5 Ryan Haddad 2021-02-11 12:56:07 PST 
Comment on attachment 420026 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=420026&action=review

> LayoutTests/platform/mac-wk2/TestExpectations:1329
> +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]

Let's mark this BigSur+, since presumably it wold affect future releases too.

> LayoutTests/platform/mac-wk2/TestExpectations:1331
> +

Nit: extra newline here
Comment 6 Robert Jenner 2021-02-11 15:59:50 PST 
Created attachment 420059 [details]
Patch
Comment 7 Robert Jenner 2021-02-11 16:04:12 PST 
(In reply to Ryan Haddad from comment #5)
> Comment on attachment 420026 [details]
> Patch
> 
> View in context:
> https://bugs.webkit.org/attachment.cgi?id=420026&action=review
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1329
> > +webkit.org/b/221218 [ BigSur ] imported/w3c/web-platform-tests/media-source/mediasource-config-change-webm-v-framesize.html [ Pass Failure ]
> 
> Let's mark this BigSur+, since presumably it wold affect future releases too.
> 
> > LayoutTests/platform/mac-wk2/TestExpectations:1331
> > +
> 
> Nit: extra newline here

Changes made, how does it look now?
Comment 8 Robert Jenner 2021-02-11 16:26:50 PST 
Created attachment 420062 [details]
Patch
Comment 9 Robert Jenner 2021-02-17 16:13:49 PST 
It does appear to be flakey failing in wk1 as well. Updating title and patch.
Comment 10 Robert Jenner 2021-02-17 16:20:37 PST 
Created attachment 420754 [details]
Update TestExpectations
Comment 11 Truitt Savell 2021-02-17 16:22:34 PST 
Comment on attachment 420754 [details]
Update TestExpectations

Clearing flags on attachment: 420754

Committed r273045 (234246@main): <https://commits.webkit.org/234246@main>
Comment 12 Ryan Haddad 2021-03-22 14:48:21 PDT 
The test is consistently failing on Big Sur WK1 with the following diff:
-PASS Tests webm video-only frame size changes.
+FAIL Tests webm video-only frame size changes. assert_true: video/webm;codecs="vp8" is supported. expected true got false

perhaps that is expected, though.

I don't see any failures on Big Sur WK2 since 02/09/21: https://results.webkit.org/?suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmedia-source%2Fmediasource-config-change-webm-v-framesize.html&flavor=wk2&platform=mac

We may be able to remove the expectation for WK2 and close this one out.
Comment 13 Ben Schwartz 2024-06-03 16:58:56 PDT 
This no longer appears to be occurring. Removing test expectations. Please file a new bug if issues persist.

Tracking test expectations removal in bug #274990 (rdar://129080660).