RESOLVED FIXED 230617
gl.texImage2D upload of getUserMedia streams via <video> element fails 
https://bugs.webkit.org/show_bug.cgi?id=230617
Summary gl.texImage2D upload of getUserMedia streams via <video> element fails
connell
Reported 2021-09-22 05:59:04 PDT
Uploading a camera stream to a WebGL texture using getUserMedia(...), a <video> element, and gl.texImage2D(...) is failing in Safari on the iOS 15.1 beta 1 release. It works correctly on the release version of iOS 15.0 and earlier. I've prepared a minimal test case here: https://clv.zappar.io/3571850569055456732/1.0.0/ The page should show a live camera feed in the canvas after allowing camera permissions. On iOS 15.1 beta 1 no camera feed appears. The source for this test case is entirely within the page itself should you wish to take a look at how it's implemented. This is a common flow for web-based augmented reality and image processing applications. We have many sites and customers who rely on this behaviour to work correctly, so this is a major regression and concern for us. The same issue is also reported in the following bug but I've filed this one with a minimal test case and a more specific title in the hopes that it helps surface this significant regression to the right team :-) https://bugs.webkit.org/show_bug.cgi?id=230589 Let me know if I can help with any questions!
Attachments
Patch (52.27 KB, patch)
2021-09-24 08:39 PDT, Kimmo Kinnunen 		ews-feeder: commit-queue-
DetailsFormatted DiffDiff
Patch (14.27 KB, patch)
2022-01-14 04:22 PST, Kimmo Kinnunen 		no flags
DetailsFormatted DiffDiff
View All Add attachment proposed patch, testcase, etc.
Radar WebKit Bug Importer
Comment 1 2021-09-22 10:45:26 PDT
<rdar://problem/83407577>
Dean Jackson
Comment 2 2021-09-22 11:37:39 PDT
*** Bug 230589 has been marked as a duplicate of this bug. ***
Brent Fulgham
Comment 3 2021-09-22 11:43:48 PDT
It looks like this regressed in Bug 228821.
Simon Taylor
Comment 4 2021-09-23 02:51:38 PDT
Thanks Brett. I see the commit in that bug references it affecting the CPU codepath only - @kkinnunen is this perhaps another case like in 216259 and 215908 where some change has forced these uploads down the CPU path rather than the GPU one?
Kimmo Kinnunen
Comment 5 2021-09-24 08:39:52 PDT
Created attachment 439150 [details] Patch
Eric Carlson
Comment 6 2021-09-24 09:03:37 PDT
Comment on attachment 439150 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review > Source/WebCore/html/HTMLVideoElement.cpp:-318 > -RefPtr<NativeImage> HTMLVideoElement::nativeImageForCurrentTime() > -{ > - if (!player()) > - return nullptr; > - > - return player()->nativeImageForCurrentTime(); > -} > - I would rather not remove this as I think MediaPlayer should be an implementation detail, and I'd like to make it private. > Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559 > + if (m_lastImage) > + return std::nullopt; You return null if there *is* an image?
Kenneth Russell
Comment 7 2021-09-24 11:00:19 PDT
Comment on attachment 439150 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review >> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559 >> + return std::nullopt; > > You return null if there *is* an image? Looks like a typo compared to MediaPlayerPrivateMediaStreamAVFObjC - maybe the cause of the widespread test failures.
Kenneth Russell
Comment 8 2021-09-24 11:00:28 PDT
Comment on attachment 439150 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=439150&action=review >> Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2559 >> + return std::nullopt; > > You return null if there *is* an image? Looks like a typo compared to MediaPlayerPrivateMediaStreamAVFObjC - maybe the cause of the widespread test failures.
Kenneth Russell
Comment 9 2021-09-24 11:01:32 PDT
The new accelerated code paths added in this patch look very cool though!
connell
Comment 10 2021-09-29 07:39:51 PDT
Just a note to say that we still see this issue on iOS 15.1 Beta 2.
Brent Fulgham
Comment 11 2021-09-29 08:55:59 PDT
(In reply to connell from comment #10) > Just a note to say that we still see this issue on iOS 15.1 Beta 2. I believe this will be part of Beta 3.
Kimmo Kinnunen
Comment 12 2021-09-29 22:36:25 PDT
*** Bug 230879 has been marked as a duplicate of this bug. ***
Kimmo Kinnunen
Comment 13 2022-01-13 05:11:59 PST
Regression was fixed by reverting r280963 (Bug 228821) for iOS 15.2 (rdar://83587220) This bug still persists in trunk. Repurposing this bug to fix in trunk.
Kimmo Kinnunen
Comment 14 2022-01-14 04:22:31 PST
Created attachment 449160 [details] Patch
youenn fablet
Comment 15 2022-01-14 06:15:19 PST
Comment on attachment 449160 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=449160&action=review > Source/WebCore/ChangeLog:9 > + Fix MSE camera to WebGL texture uploads. s/MSE/MediaStreamTrack/ > Source/WebCore/ChangeLog:12 > + Turns out MSE does not have these implemented, so currently fall back It should be easy to add such methods. We have m_currentVideoSample we can grab for that using a lock.
EWS
Comment 16 2022-01-14 12:42:29 PST
Committed r288025 (246051@main): <https://commits.webkit.org/246051@main> All reviewed patches have been landed. Closing bug and clearing flags on attachment 449160 [details].
rigel
Comment 17 2022-01-29 21:10:11 PST
I'm currently seeing this behavior (black screen, no camera feed) on iOS 15.4b1 (iPhone 11) on all 8th Wall and Zappar websites, including the test case listed at the top: https://clv.zappar.io/3571850569055456732/1.0.0/
Note You need to log in before you can comment on or make changes to this bug.