Bug 232076

Cross referencing this older issue for some additional background - https://bugs.webkit.org/show_bug.cgi?id=211295

<rdar://problem/84506557>

I suspect this is related to the video decoder being moved to the GPU Process in iOS 15, which had some intended consequences for the codepath used when the video source is in a blob / data URI. We were able to find a workaround of sorts, but it's an ugly one: If the video data is provided by a Service Worker then it seems to play back with normal performance and without the initial massive latency. Roughly the steps: 1) Spin up a Service Worker 2) postMessage to the worker with a blob of data and a unique (fake) URL to be used for the video 3) In the worker, add the data to the cache using the unique URL 4) Set the video's src to the path that the Service Worker will respond to The service worker will need to support range requests - we used workbox to provide that side of the implementation. I'll attach the service-worker side of things in case it helps anyone else.

(In reply to Simon Taylor from comment #3) > which had some intended consequences for the codepath I meant *unintended*, of course...

Created attachment 442151 [details] Example service worker that offers an (extremely ugly) workaround An example service worker implementation that provides an ugly workaround to this bug. From the main page, you post a blob over to the worker with the URL you want to use for it like this: navigator.serviceWorker.ready.then(reg => { if (reg.active) { reg.active.postMessage({source: "/__video__[uniquepath]", blob }); } }); Then wait for the message back to know it's been added, and set the source to the path you used when posting the data across.

Hi Simon! This workaround is definitely a legit thing, however, it only applies for the cases when you *can use* a service worker. In our scenario we have to have a single HTML page (no external resources, period) which stops us from using service workers. P.S. If I am wrong and there's a way to spin up a service worker from inline code (just like using Data URI for Web Worker) – please lmk, we are eager to try any workaround possible.

I think you're out of luck in that case, looks like service workers can't be specified inline. I guess that makes sense from a security POV - prevents third party js from injecting a service worker to man-in-the-middle all your network traffic. Back on the bug, there's a demo page here that shows videos with data uris for the source: https://iandevlin.com/html5/data-uri/video.php Using a blob via an object URL seems to have similar performance to me. One weird finding, switching to an <img> tag with the mp4 as src ( as described in https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari ) appears to give much better performance, and that would work inline. Downside is no audio, it loops, and there's no controls / timeline or anything. Depending on your use case it might be helpful though...

Created attachment 442160 [details] Screen recording showing large delay before start Screen recording showing the large performance difference between data uris when specified as a <video> source or an <img> src. The very long time (10s +) between pressing play and the video starting can be seen in the first page - note the progress bar also takes a long time to complete. I've removed the web source from the demo page I linked above, so both of these pages are 768KB and just include the mp4 inline.

> One weird finding, switching to an <img> tag with the mp4 as src ( as described in https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari ) appears to give much better performance, and that would work inline. Downside is no audio, it loops, and there's no controls / timeline or anything. Depending on your use case it might be helpful though... Yes you are right - I mentioned this in observations in bug description as well. > The very long time (10s +) between pressing play and the video starting can be seen in the first page - note the progress bar also takes a long time to complete. Yeah I guess the loading bar is showing until the video is read in full (all 200 times it needs to due to random seeks across the file).

Created attachment 442162 [details] Demo page with video element and data uri

Created attachment 442163 [details] Demo page with img element and data uri source

(In reply to Anton Zhuravsky from comment #9) > > One weird finding, switching to an <img> tag with the mp4 as src ( as described in https://developer.apple.com/documentation/webkit/delivering_video_content_for_safari ) appears to give much better performance, and that would work inline. Downside is no audio, it loops, and there's no controls / timeline or anything. Depending on your use case it might be helpful though... > > Yes you are right - I mentioned this in observations in bug description as > well. Ha, so you did, sorry for missing that! I've just tried toggling the Experimental Setting for GPU Process: Media to off (and rebooted), and still see the same horrible performance, so my hunch that had something to do with the regression seems probably wrong. Over to Apple... (ps: if you're able to change the title, can you add a mention of blob / object URL too)

In our production implementation (https://my.panomoments.com/explore) we, and have been able to load videos as blobs in both MacOS, iOS, and iPadOS. However, performance has been significantly degraded recently on iOS, iPadOS, and MacOS (see #229413), and especially when using 100% I-Frame videos (see #211295). For me, using the service worker code above actually results in worse performance when using both our 100% I-Frame design, and our preferred Safari optimized design (which also uses blob videos, but not 100% I-Frames). So we're kind of stuck in a bad spot across all Apple platforms...

Further, when testing the service worker workaround on iPadOS 15.1, it throws this error: Unhandled Promise Rejection: TypeError: undefined is not an object (evaluating 'navigator.serviceWorker.addEventListener') I haven't tested on iOS, but I imagine it might do the same. MacOS Safari seems to work fine with the service worker, but with worse performance than without the service worker.

Hi Dustin, The Service Worker approach is definitely working for us in production on iOS 15 and iPad OS 15. It's the only way we've found to avoid the massive delay (10s or more) in playback starting (accompanied by very high CPU and memory usage). Is your non-worker Blob solution not suffering those effects? We're using it for more "normal" resolution files - 10MBish, 720p or so - direct blob or data uri approaches were fine in iOS 14 but unusable in iOS 15, the Service Worker approach did get those videos to work OK for us. I imagine there's still some extra overhead from having JS involved in the network request path though.

It's possible the delay on initial load is less when using the service worker, but as my implementation requires constant seeking, the performance characteristics may be very different than your use-case (which I believe is just normal video playback).

Howdy folks, a quick update on our investigation into this bug. It appears a regression in a framework used for loading data through custom URL schemes occurred in iOS 15 where the "Range:" header was not included in blob:// URL requests. This caused the entire contents of the Blob to be fetched, rather than just the few bytes requested by the media framework. While the underlying issue is being investigated, I'm working on a workaround in bug #232195.

Hey Jer! Thank you very much for looking into this. Just my 5 cents here: when a video file is supplied as Data URI we _can_ actually see Accept: Range requests being executed against Data URI (whilst providing the blob to the video element doesn't show these headers; yet both do exhibit hundreds of HTTP requests reading video data). Prior to iOS 15, there were no repetitive HTTP requests to neither Data URI nor blobs, so from the outside it looks like a problem with video data handler, not HTTP stack (since both Data URI and blobs are in memory, it makes little to no sense to request separate regions of the data via HTTP given the video player has access to the whole chunk at once).

Jer, And just a quick follow-up to Anton's note above - Per https://bugs.webkit.org/show_bug.cgi?id=211295 - this issue gets significantly worse with videos that contain lots of I-Frames (and got worse with the more recent regression). It would be awesome if a potential fix for 232076 could also help out 211295. Let me know if you have any questions or would like a simple replication test. Thanks!

Dustin, I _think_ iframe-only videos are no different w.r.t the underlying problem – it's getting worse simply because the size of the video is bigger, leading to more memory consumption and faster exhaustion overall.

(In reply to Anton Zhuravsky from comment #20) > Dustin, > > I _think_ iframe-only videos are no different w.r.t the underlying problem – > it's getting worse simply because the size of the video is bigger, leading > to more memory consumption and faster exhaustion overall. Agreed; it's likely that it's a problem with larger files (and more I-frames, more Kb/s, more problems).

This issue was mitigated in Bug 232195. This should now work properly.

Just wanted to add, that the service worker workaround also does not work for WKWebViews, as they no not support service workers. Creating Blob URLs and inserting them into video tags is a very common usecase and has to be supported by any browser..

(In reply to maxcodefaster from comment #23) > Just wanted to add, that the service worker workaround also does not work > for WKWebViews, as they no not support service workers. Creating Blob URLs > and inserting them into video tags is a very common usecase and has to be > supported by any browser.. That’s not quite accurate. Applications using WKWebViews can use service workers if they opt into “app-bound” domains, as covered at WWDC and in our blog posts. Applications that register and are approved to use the default web browser entitlement are also able to use service workers.

(In reply to Brent Fulgham from comment #24) > (In reply to maxcodefaster from comment #23) > > Just wanted to add, that the service worker workaround also does not work > > for WKWebViews, as they no not support service workers. Creating Blob URLs > > and inserting them into video tags is a very common usecase and has to be > > supported by any browser.. > > That’s not quite accurate. Applications using WKWebViews can use service > workers if they opt into “app-bound” domains, as covered at WWDC and in our > blog posts. > > Applications that register and are approved to use the default web browser > entitlement are also able to use service workers. Sorry my fault. You are right. I forgot to add, that Service Workers do not work in a WKWebView with a custom schema other than https:// or http://. This is the case for hybrid frameworks like Capacitor / Cordova.

Hi everyone, I was able to test this on a first generation iPad Pro running iPad OS 15.4 and on an iPhone X on iOS 15.4. Upon initial testing it looked like this fix finally addressed the issue. However, upon further testing I noticed that playback/seeking would sometimes stall (usually around the same place in the video). While this could be a different issue, it's interesting that the one piece of valuable debug I've found so far is the range request. Here is what a valid request/response looks like during working playback: Request -------- Range: bytes=4292608-4308991 Response -------- Content-Type: video/mp4 Content-Length: 16384 Content-Range: bytes 4292608-4308991/4315470 And when playback/seeking breaks, I see this malformed range request/response: Request -------- Range: bytes=4308992- Response -------- Content-Length: 6478 Content-Range No playback/seeking works after this bad 206 range request. See the full broken request/response below: Summary URL: blob:http://192.168.4.23:3000/2166e4f1-23ea-46cc-b789-c67810991241 Status: 206 Partial Content Source: Network Request User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15 Range: bytes=4308992- Referer: http://192.168.4.23:3000/u/dustinkerstein/m/both Origin: http://192.168.4.23:3000 Accept: */* X-Playback-Session-Id: 7C8DCC6C-1022-4E7D-9A17-0DFF975BE68C Response Content-Type: video/mp4 Content-Length: 6478 Content-Range Here is a simple Javascript only test-case which seeks forwards and backwards in the video - https://jsfiddle.net/dustinkerstein/o9fy1483 - On my iPad Pro playback breaks when it starts to seeks backwards and freezes on the first or second frame (and this is where the last broken range request is found). Does this seem like a related issue, or possibly something else? Regardless, this bug will continue to prevent our company from being able to use blobs as video source due to our reliance on working seek functionality.

(In reply to Dustin Kerstein from comment #26) > Hi everyone, > > I was able to test this on a first generation iPad Pro running iPad OS 15.4 > and on an iPhone X on iOS 15.4. Upon initial testing it looked like this fix > finally addressed the issue. However, upon further testing I noticed that > playback/seeking would sometimes stall (usually around the same place in the > video). While this could be a different issue, it's interesting that the one > piece of valuable debug I've found so far is the range request. Here is what > a valid request/response looks like during working playback: > > Request > -------- > Range: bytes=4292608-4308991 > > Response > -------- > Content-Type: video/mp4 > Content-Length: 16384 > Content-Range: bytes 4292608-4308991/4315470 > > And when playback/seeking breaks, I see this malformed range > request/response: > > Request > -------- > Range: bytes=4308992- > > Response > -------- > Content-Length: 6478 > Content-Range > > No playback/seeking works after this bad 206 range request. This behavior sounds like what should have been fixed by bug #238170.

That does indeed sound like similar behavior. Are you aware of any possible workarounds? I'm just a little bummed that we might need to again wait another 1,2,3+ more iOS updates before this new fix lands. Ie. The previous blob 206 fix was checked in 2021-11-03 but wasn't available to end users until ~5 months later when iOS 15.4 landed. Thanks, Dustin

Just wanted to apologize for the unhelpful attitude in my previous post (and further apologies for the extraneous email notifications). I'd be happy to help test any new code once it's available to try out. Let me know if I can be of any use. Thanks!

This isn't exactly a "fix" to the data URI problem. More of a supported workaround, really. It occurred to me that the only platform in which this issue seems to happen is coincidentally also the only one that fully supports assigning things other than a MediaStream to the HTMLMediaElement.srcObject, specifically MediaSources, Files, and most importantly: Blobs. Since Blobs are on the table, the above is a quick and painless way of getting the video to play without any obvious issues (at least, none that my testing of this has encountered). For cross-platform uses, assigning non-MediaStream objects in unsupported browsers throws an error, which you could catch and then default to assigning the Blob to the HTMLMediaElement.src property via a call to URL.createObjectURL(). Of note, a Blob as a srcObject only works if it has an appropriate Blob.type.