Bug 217263

Summary: [ iOS ] 2 mediacapture-record tests are a constant failure
Product: WebKit Reporter: Hector Lopez <hector_i_lopez>
Component: New BugsAssignee: youenn fablet <youennf>
Status: REOPENED ---    
Severity: Normal CC: webkit-bot-watchers-bugzilla, webkit-bug-importer, youennf
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=217262
Attachments:
Description Flags
Patch none

Description Hector Lopez 2020-10-02 18:05:14 PDT 
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

Tests are constant failures according to history on iOS. First occurrence of failures on iOS coincide with first testing on iOS. 

Both test were passing on macOS until r267870. From that point test imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html became flaky on macOS Catalina wk2 Debug/Release.

History:

https://results.webkit.org/?suite=layout-tests&suite=layout-tests&test=imported%2Fw3c%2Fweb-platform-tests%2Fmediacapture-record%2FMediaRecorder-no-sink.https.html&test=imported%2Fw3c%2Fweb-platform-tests%2Fmediacapture-record%2FMediaRecorder-peerconnection.https.html

Diff iOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html:

--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https-actual.txt
@@ -1,5 +1,7 @@
+
+Harness Error (TIMEOUT), message = null
 
 PASS PeerConnection MediaRecorder records audio from PeerConnection without sinks
 PASS PeerConnection MediaRecorder records video from PeerConnection without sinks
-PASS PeerConnection MediaRecorder records audio/video from PeerConnection without sinks
+TIMEOUT PeerConnection MediaRecorder records audio/video from PeerConnection without sinks Test timed out

Diff iOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

--- /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt
+++ /Volumes/Data/slave/ios-simulator-14-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-actual.txt
@@ -6,8 +6,8 @@
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format [passthrough].
-PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough].
-TIMEOUT PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough]. Test timed out
+TIMEOUT PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough]. Test timed out
+NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough].
 NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8.
 NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8.
 NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8.

Diff macOS for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html

--- /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-expected.txt
+++ /Volumes/Data/slave/catalina-release-tests-wk2/build/layout-test-results/imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https-actual.txt
@@ -1,23 +1,21 @@
 
-
-Harness Error (TIMEOUT), message = null
 
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format [passthrough].
 PASS PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format [passthrough].
-TIMEOUT PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough]. Test timed out
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9.
-NOTRUN PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9.
+PASS PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format [passthrough].
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp8. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":false,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":false} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder receives data after onstart, {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"
+FAIL PeerConnection MediaRecorder gets ondata on stopping recorded tracks {"video":true,"audio":true} with format video/webm;codecs=vp9. promise_test: Unhandled rejection with value: object "NotSupportedError: mimeType is not supported"



Reproduction for imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html on macOS Catalina wk 2 Release on r267897

run-webkit-tests imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html --iterations 1000 --exit-after-n-failures 3

Result:

Baseline search path: platform/mac-catalina -> Internal/Tests/LayoutTestsResultsForUnreleasedSoftware/mac-wk2 -> platform/mac-wk2 -> Internal/Tests/LayoutTestsResultsForUnreleasedSoftware/mac -> platform/mac -> platform/wk2 -> generic

Running 1 test

Running 1 WebKitTestRunner.

[1/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
[2/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
[3/1000] imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html failed unexpectedly (text diff)
Exiting early after 3 failures. 3 tests run.


0 tests ran as expected, 3 didn't (997 didn't run):


Regressions: Unexpected text-only failures (1)
  imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-peerconnection.https.html [ Failure ]
Comment 1 Radar WebKit Bug Importer 2020-10-02 18:05:31 PDT 
<rdar://problem/69904225>
Comment 2 Hector Lopez 2020-10-02 18:15:21 PDT 
See webkit.org/b/217262 for test expectations for Mac

Test expectations for iOS:
Comment 3 Hector Lopez 2020-10-02 18:18:36 PDT 
Test expectations for iOS:

https://trac.webkit.org/changeset/267905/webkit

See test expectation for Mac:

https://trac.webkit.org/changeset/267904/webkit
Comment 4 youenn fablet 2020-10-12 05:15:23 PDT 
Created attachment 411109 [details]
Patch
Comment 5 EWS 2020-10-12 06:55:30 PDT 
commit-queue failed to commit attachment 411109 [details] to WebKit repository.
Comment 6 EWS 2020-10-12 10:37:42 PDT 
Committed r268361: <https://trac.webkit.org/changeset/268361>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 411109 [details].
Comment 7 youenn fablet 2020-10-12 23:40:28 PDT 
Reopening for further investigation
Comment 8 Hector Lopez 2020-10-20 09:42:35 PDT 
imported/w3c/web-platform-tests/mediacapture-record/MediaRecorder-no-sink.https.html

Test expectation adjustment:

https://trac.webkit.org/changeset/268737/webkit