Bug 206843 - Can't draw stream->video->texture->canvas without also drawing video.
Summary: Can't draw stream->video->texture->canvas without also drawing video.
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 13
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-01-27 13:56 PST by Nicholas Butko
Modified: 2020-01-28 05:21 PST (History)
5 users (show)

See Also:


Attachments
source code for repro case (5.05 KB, text/javascript)
2020-01-27 13:56 PST, Nicholas Butko
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicholas Butko 2020-01-27 13:56:57 PST
Created attachment 388902 [details]
source code for repro case

WebKit appears to stop streaming the camera feed to a <video> tag when the element is not visible, either because it is display none, or it is fully occluded by other DOM elements. Streaming to an off-screen <video> is common when drawing the camera feed to a texture for canvas display, for example to apply visual effects on top of the camera feed, or to synchronize display of the camera feed with the result of some visual processing (QR code detection, image target tracking, SLAM, etc.).

Our workaround solution is to make the camera feed visible but all-but-hidden, e.g. by displaying as a block element behind the drawing canvas, offset by one pixel:

  const setVideoForSafari = ({video, canvas}) => {
    if (!canvas.style['z-index']) {
      canvas.style['z-index'] = 1
    }
    canvas.style.top = '1px'
    canvas.style.left = '0px'
    canvas.style.position = 'absolute'
    video.setAttribute('playsinline', true)
    video.style.display = 'block'
    video.style['z-index'] = canvas.style['z-index'] - 1
    video.style.top = '0px'
    video.style.left = '0px'
    video.style.position = 'absolute'
    video.style.opacity = '0.01'
    video.parentNode.style.position = 'absolute'
  }

This approach adds inefficiency because camera pixels are drawn twice, and adds a subtle visual defect because the camera feed is one-pixel visible. This workaround was required for WebAR activations by Pink Floyd, BBC America's Dr Who, Bailey's, Jumanji, Johnnie Walker, Game of Thrones, Monopoly, Ally Bank, Adidas, Porsche, Eva Air, London's Waterloo Station, Spider-Man: Far From Home, The Phillies, Lego, General Mills, Red Bull, Toyota, and many more.

A basic repro case can be viewed on an iPhone here: 8th.io/k4c5d

The repro app does the following:
* getUserMedia -> stream -> video
* iOS only: set the video style to be display: block, z-index: canvas.style.zIndex - 1, 1 pixel offset from canvas
* video -> texture (there is a rotating pool of 10 textures)
* texture -> sepia shader -> canvas -> window
* on a button press, toggle video.style.display to block / none

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.
Comment 1 Radar WebKit Bug Importer 2020-01-27 22:28:34 PST
<rdar://problem/58946401>