Bug 218637

I can reproduce this in iPhone 12 Pro.

<rdar://problem/71102126>

On iPhone 12 fails with: Asked to copy an unsupported pixel format ('&8v0'). when interpreting the result of CVPixelBufferGetPixelFormatType. The supported ones are '420v' and '420f'. Unsure if the above pixel format is real fourcc format or just misinterpretation of OSType enumeration.

And to clarify: Not all WebGL video copies fail. It appears that at least the bunny one fails, but at least one other test case in bug 215908 succeeds

Kimmo, since you've triaged this already could you officially take it? Unfortunately James and I can't feasibly debug anything that happens only on real iOS hardware.

Created attachment 413929 [details] Patch

Comment on attachment 413929 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=413929&action=review Very nice and well thought through fix. It looks good to me, especially given you've verified the fix on hardware - two small comments above. However I think it would be best if jer.noble reviewed this too and approved it. > Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2216 > + (__bridge NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{ /*empty dictionary*/ }, Just checking - it's okay to specify this property for all video playback? > Source/WebCore/platform/graphics/avfoundation/objc/MediaPlayerPrivateAVFoundationObjC.mm:2363 > + CVPixelBufferRef pixelBuffer = updateLastPixelBufferForWebGL(); One note - the software upload path into GraphicsContext goes through the call chain paintCurrentFrameInContext -> paintWithVideoOutput -> updateLastImage, which calls updateLastPixelBuffer. However, this doesn't have the same requirements on the pixel buffer format that copyVideoTextureToPlatformTexture does, so that call to updateLastPixelBuffer doesn't need to be updated to updateLastPixelBufferForWebGL.

Comment on attachment 413929 [details] Patch Since other reviewers seem busy, let me set r+ since you've tested this locally and it fixes a major regression. Please set cq? if you'd like me to cq+ it too.

Created attachment 414198 [details] Patch

Created attachment 414199 [details] Patch

(In reply to Kenneth Russell from comment #8) > Comment on attachment 413929 [details] > Patch > > Since other reviewers seem busy, let me set r+ since you've tested this > locally and it fixes a major regression. Please set cq? if you'd like me to > cq+ it too. Thanks for the review. Jer explained to me that using PixelBufferTransferSession is a software fallback. I moved the implementation to bug 218971 where we should improve the software fallback, since currently it's not really working (contributing to two regressions in iOS releases). I uploaded Jer's patch for treating the currently frequent-but-not-working format similar to a format we already support.

Committed r269893: <https://trac.webkit.org/changeset/269893> All reviewed patches have been landed. Closing bug and clearing flags on attachment 414199 [details].

I'm still seeing this behaviour on the latest iOS 14.4 (18D52) on iPhone 12 Mini. I know Apple doesn't comment on release plans but just wanted to mention again as this is a pretty major regression that has persisted for multiple public releases since the webkit patch landed and this bug was closed. Kimmo - a couple of practical questions if I may: (In reply to Kimmo Kinnunen from comment #4) > And to clarify: > Not all WebGL video copies fail. It appears that at least the bunny one > fails, but at least one other test case in bug 215908 succeeds This is interesting, as it suggests a change in the encoding may allow HLS videos to work in current iOS releases. From looking at the patches it seems the video decoder on the iPhone 12 produces frames with a new internal format, so I'd be surprised if the behaviour was different with different videos, but perhaps there's a resolution threshold where it switches output format? Any further hints appreciated... Related to Bug 216250 (the performance regression) - I assume even videos that do work on iPhone 12 series are using the software fallback? Performance numbers strongly indicate that's the case, but as this hardware has always shipped with at least iOS 14 it's hard to know for sure.

The bug still present in the iOS 14.5 beta 2 (18E5154f), released a couple of days ago (Feb 16).

Thanks, I'll inspect this the coming week.

Reopening to attach new patch.

Created attachment 421397 [details] Patch

Note that there are important steps to take when updating ANGLE. See https://trac.webkit.org/wiki/UpdatingANGLE

Thanks for the report, sorry it took so long. Repened this as per the effect of the problem is the same.

Tracking under <rdar://problem/74702474>

Comment on attachment 421397 [details] Patch View in context: https://bugs.webkit.org/attachment.cgi?id=421397&action=review Looks good. r+ > Source/ThirdParty/ANGLE/src/libANGLE/renderer/gl/eagl/IOSurfaceSurfaceEAGL.mm:321 > + // However, there caller might supply us non-public pixel format, which makes exhaustive checks Should this be: there caller -> the caller ? > Source/ThirdParty/ANGLE/src/libANGLE/renderer/gl/eagl/IOSurfaceSurfaceEAGL.mm:322 > + // problematic. Can any validation be done at all? What are the consequences of failing this validation? It's important to not expose uninitialized GPU memory, etc.

Created attachment 421524 [details] Patch

(In reply to Kenneth Russell from comment #21) > Can any validation be done at all? What are the consequences of failing this > validation? It's important to not expose uninitialized GPU memory, etc. I'll try to ask what kind of checks we should do, e.g. what is the impact of passing internalFormat, format, type to texImageIOSurface

Committed r273582: <https://commits.webkit.org/r273582> All reviewed patches have been landed. Closing bug and clearing flags on attachment 421524 [details].

Thanks for the fix. Since this issue has been pending for a few months, is there any possibility of ensuring/expediting this fix to be included in the next release? We have multiple clients blocked and waiting for this to be resolved so any visibility would be appreciated.

I'm unable to confirm this issue is resolved. I can reproduce the issue on iPhone 12 iOS 14.5 Beta. @Kimmo Kinnunen I'm not really sure why this issue has status Resolved Fixed. It is not resolved.

(In reply to Mateusz Śpiewak from comment #26) > I'm unable to confirm this issue is resolved. I can reproduce the issue on > iPhone 12 iOS 14.5 Beta. > > @Kimmo Kinnunen I'm not really sure why this issue has status Resolved > Fixed. It is not resolved. The issue is fixed in the WebKit development source repository. This will be released in iOS Safari in a future iOS release. Unfortunately I cannot give a timeline when or in which public beta this can be tested.

(In reply to Kimmo Kinnunen from comment #27) > (In reply to Mateusz Śpiewak from comment #26) > > I'm unable to confirm this issue is resolved. I can reproduce the issue on > > iPhone 12 iOS 14.5 Beta. > > > > @Kimmo Kinnunen I'm not really sure why this issue has status Resolved > > Fixed. It is not resolved. > > The issue is fixed in the WebKit development source repository. This will be > released in iOS Safari in a future iOS release. Unfortunately I cannot give > a timeline when or in which public beta this can be tested. @Kimmo Kinnunen Sorry to bug you again but it's been another month and was wondering if there's any information you could share regarding timelines. The issue itself has been pending for months and we have multiple clients blocked and waiting for this to be resolved so any visibility would be appreciated.

(In reply to Taeheon Kim from comment #28) > (In reply to Kimmo Kinnunen from comment #27) > > (In reply to Mateusz Śpiewak from comment #26) > > > I'm unable to confirm this issue is resolved. I can reproduce the issue on > > > iPhone 12 iOS 14.5 Beta. > > > > > > @Kimmo Kinnunen I'm not really sure why this issue has status Resolved > > > Fixed. It is not resolved. > > > > The issue is fixed in the WebKit development source repository. This will be > > released in iOS Safari in a future iOS release. Unfortunately I cannot give > > a timeline when or in which public beta this can be tested. > > @Kimmo Kinnunen Sorry to bug you again but it's been another month and was > wondering if there's any information you could share regarding timelines. > The issue itself has been pending for months and we have multiple clients > blocked and waiting for this to be resolved so any visibility would be > appreciated. Apple does not discuss release schedules or plans for future updates. We will notify you when a public build is available with such a fix.

Hi Kimmo - Hannah Bianchini, Executive Producer of Microsoft Mixed Reality Capture Studios. Do you know if there's a manager on the Safari team that I can escalate this issue to? I'd like to find a way to unblock a number of clients. Thanks Hannah@microsoft.com

(In reply to Hannah Bianchini from comment #30) > Hi Kimmo - > > Hannah Bianchini, Executive Producer of Microsoft Mixed Reality Capture > Studios. Do you know if there's a manager on the Safari team that I can > escalate this issue to? I'd like to find a way to unblock a number of > clients. > > Thanks > Hannah@microsoft.com Hannah, I would recommend you reach out to the Microsoft business relationship contact at Apple. The WebKit Open Source project is not really the right venue for these discussions as it has no ability to decide what bug fixes go into which release, nor the ability to share details about upcoming releases. Microsoft's existing communications channels with Apple are designed for these kinds of discussions, and will likely get you better information than posting to this bug.

I just checked, and I believe the fix is part of iOS 14.6 Beta 2.

Ok, I'll try and get an Iphone 12 to confirm but won't be able to install beta on it to test. I think the original regression was affecting Iphone 12. I'll have to let people know. So within another 6 months to get 14.6 seeing 14.5 was just released ?

(In reply to Daniel Rossi from comment #33) > Ok, I'll try and get an Iphone 12 to confirm but won't be able to install > beta on it to test. I think the original regression was affecting Iphone 12. > I'll have to let people know. So within another 6 months to get 14.6 seeing > 14.5 was just released ? Hopefully you noticed, but iOS 14.6 was released on 5/24, so less than a week after your message.

This was ages ago. I am seeing first frame HLS texture render issues on IOS 15 still and its random. Page reload will show it. Resizing the canvas on a resize event doesn't help anymore. I've been dealing with this bug and finding work arounds since IOS 10 so going to just leave it at that. It might be good if webkit had a HLS texture as part of their webgl tests. I found when reporting they don't.