Bug 207128 - Drawing canvas to texture or canvas2d sometimes fails, flashes white
Summary: Drawing canvas to texture or canvas2d sometimes fails, flashes white
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: Canvas (show other bugs)
Version: Safari 13
Hardware: Unspecified Unspecified
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-02-03 10:01 PST by Nicholas Butko
Modified: 2020-02-04 09:55 PST (History)
9 users (show)

See Also:


Attachments
source code for repro case (8.11 KB, text/javascript)
2020-02-03 10:01 PST, Nicholas Butko
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicholas Butko 2020-02-03 10:01:34 PST
Created attachment 389533 [details]
source code for repro case

WebKit regularly appears to fail to draw canvas data to another source, and when this happens the source canvas flashes white. We have observed this behavior when drawing from canvas to canvas:

   captureCanvas_.getContext('2d').drawImage(srcCanvas_, ...)

and we have also seen the same behavior with populating a different texture in the same WebGlContext with texImage2D, e.g.

   gl.bindTexture(gl.TEXTURE_2D, captureTex)
   gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, canvas)

When drawing the canvas fails, the destination image is black, including the alpha channel. One workaround solution is to detect this transparent black image and retry capture until we have image data, as in this example:

  const CAPTURE_HEIGHT = 1280
  const REPEAT_TRIES = 4
  let captureCanvas_ = document.createElement('canvas')
  let captureContext_ = captureCanvas_.getContext('2d')
  captureCanvas_.width = CAPTURE_HEIGHT * 3 / 4
  captureCanvas_.height = CAPTURE_HEIGHT

  // Returns base64 string of image data upon success, null upon failure
  const getCanvasData = (canvas) => {
    captureContext_.drawImage(canvas, 0, 0, CAPTURE_HEIGHT * 3 / 4, CAPTURE_HEIGHT)

    // drawImage can fail on iOS leaving the canvas transparent. We return null so we can retry.
    const firstPixel = captureContext_.getImageData(0, 0, 1, 1)
    if (firstPixel.data[3] === 0) {
      return null
    }
    return captureContext_.getImageData(0, 0, 1, 1)
      ? null
      : captureCanvas_.toDataURL('image/jpeg', JPG_COMPRESSION / 100)
          .substring('data:image/jpeg;base64,'.length)
  }

  const takeScreenshot = (canvas, repeatTries = 0) => new Promise((resolve, reject) => 
    window.requestAnimationFrame(() => {
      const imageData = getCanvasData(canvas)
      if (repeatTries < MAX_REPEAT_TRIES && !imageData) {
        setTimeout(() => takeScreenshot(canvas, repeatTries + 1).then(resolve, reject), 60)
      } else {
        if (imageData) {
          updateStatusText(`OK after ${repeatTries} tries.`)
          resolve(imageData)
        } else {
          updateStatusText(`Failed ${repeatTries} times.`)
          reject(new Error('Unable to read pixels from canvas.'))
        }
      }
    })
  )

This workaround is imperfect for a number of reasons:
 * It adds expense to read back pixel values to perform the test.
 * Whenever drawing fails, the canvas flashes white, leading to a strobing effect on one or more retries.  To work around this, in production, we always need to show a full screen white div that fades out after capture succeeds or ultimately fails so that at least the effect appears intentional.
 * Capturing multiple frames in sequence (for example to capture a gif) gives inconsistent frame delay, leading to a jumpy video with poor quality.
 * Sometimes capture fails even after 10 tries, and the user is ultimately unable to receive an image, which is a poor and confusing experience.

Capturing and sharing augmented image content is a key flow for user engagement. This workaround was required for WebAR activations by Pink Floyd, Bailey's, Nike, Johnnie Walker, Game of Thrones, Monopoly, Ally Bank, Adidas, Porsche,  Spider-Man: Far From Home, The Phillies, Lego, Red Bull, Toyota, Coach and many more.

A basic repro case can be viewed on an iPhone here: 8th.io/3usye

The repro app does the following:
* getUserMedia -> stream -> video
* video -> texture (there is a rotating pool of 10 textures)
* texture -> sepia shader -> canvas -> window
* on a button press, call takeScreenshot(canvas) as above
* print the number of retries before successful capture, or a failure message if capture failed.

We tested this with an iPhone11 Pro with iOS 13.3.

Source code for the repro is attached.
Comment 1 Radar WebKit Bug Importer 2020-02-04 09:30:39 PST
<rdar://problem/59151627>