Bug 214689 - getUserMedia aspectRatio overconstrained in portrait orientation
Summary: getUserMedia aspectRatio overconstrained in portrait orientation
Status: NEW
Alias: None
Product: WebKit
Classification: Unclassified
Component: WebRTC (show other bugs)
Version: Safari 13
Hardware: iPhone / iPad iOS 13
: P2 Normal
Assignee: Nobody
URL:
Keywords: InRadar
Depends on:
Blocks:
 
Reported: 2020-07-23 09:23 PDT by Kyle Dunn
Modified: 2020-07-24 09:30 PDT (History)
2 users (show)

See Also:


Attachments
Demo of aspect ratio bug in portrait mode (1.93 KB, text/html)
2020-07-23 09:23 PDT, Kyle Dunn
no flags Details
applyConstraint Workaround Test (2.15 KB, text/html)
2020-07-24 09:03 PDT, Kyle Dunn
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Kyle Dunn 2020-07-23 09:23:31 PDT
Created attachment 405046 [details]
Demo of aspect ratio bug in portrait mode

I have a use case that requires me to get a 4:3 video from getUserMedia on iPad / iOS devices. I can do this reliably when the device is in landscape by setting the `aspectRatio` constraint to 4/3 but when the device is in portrait (thus requiring a relative 3/4 aspect ratio) an `OverconstrainedError: Invalid constraint` error is thrown.

If I specify exact dimensions, e.g. 640x480 in landscape or 480x640 in portrait, I am able to obtain a 4:3 video regardless of orientation but this requires exact dimensions which is problematic when trying to obtain a maximum camera resolution that may or may not be supported by the device.

I am using an iPad Pro (12.9-inch) (3rd generation) on iPadOS 13.6 and I have attached a demo to reproduce the issue when tested on a mobile device in portrait mode.
Comment 1 Radar WebKit Bug Importer 2020-07-23 16:34:59 PDT
<rdar://problem/66020174>
Comment 2 youenn fablet 2020-07-24 07:23:33 PDT
I confirmed the issue.
Comment 3 youenn fablet 2020-07-24 07:34:13 PDT
In general, I would recommend to use ideal constraints for getUserMedia.
Then, you can check the results and use applyConstraints to resize at will.

Would the following workaround work for you:
- Call getUserMedia with aspectRatio 4/3 as ideal constraints whatever the orientation.
- Use applyConstraints to flip height and width in case of portrait mode.
Comment 4 youenn fablet 2020-07-24 07:39:53 PDT
In case we can resize frames at will, we could also potentially have arbitrary aspect ratio as well.
Comment 5 Kyle Dunn 2020-07-24 09:03:15 PDT
Created attachment 405149 [details]
applyConstraint Workaround Test

(In reply to youenn fablet from comment #3)
> In general, I would recommend to use ideal constraints for getUserMedia.
> Then, you can check the results and use applyConstraints to resize at will.
> 
> Would the following workaround work for you:
> - Call getUserMedia with aspectRatio 4/3 as ideal constraints whatever the
> orientation.
> - Use applyConstraints to flip height and width in case of portrait mode.

Thanks for the quick response/workaround. I'm seeing odd behavior when initially applying constraints in portrait mode - I have to flip the height/width with applyConstraints multiple times before it actually crops the video. This does not seem to be an issue in landscape. The dimensions set with applyConstraints do get reflected in getSettings, but only reliably crop the video after "flipping" 3-5 times. However, once the video is reliably cropping it works exactly as expected and would be an acceptable workaround. 

A new demo with this workaround is attached.
Comment 6 youenn fablet 2020-07-24 09:18:22 PDT
https://bugs.webkit.org/show_bug.cgi?id=214741 should allow to set an aspect ratio of 3/4.
Let's keep this bug open until we get a good idea of how we want to handle rotation.
Comment 7 youenn fablet 2020-07-24 09:19:34 PDT
(In reply to youenn fablet from comment #6)
> https://bugs.webkit.org/show_bug.cgi?id=214741 should allow to set an aspect
> ratio of 3/4.
> Let's keep this bug open until we get a good idea of how we want to handle
> rotation.

And fix the potential bug of needing to call applyConstraints multiple times.
Comment 8 Kyle Dunn 2020-07-24 09:22:35 PDT
(In reply to youenn fablet from comment #7)
> (In reply to youenn fablet from comment #6)
> > https://bugs.webkit.org/show_bug.cgi?id=214741 should allow to set an aspect
> > ratio of 3/4.
> > Let's keep this bug open until we get a good idea of how we want to handle
> > rotation.
> 
> And fix the potential bug of needing to call applyConstraints multiple times.

Awesome, thanks Youenn!
Comment 9 youenn fablet 2020-07-24 09:23:18 PDT
> crop the video after "flipping" 3-5 times. However, once the video is

Doing your test, it seems to work for me after clicking 2 times.
Comment 10 Kyle Dunn 2020-07-24 09:30:01 PDT
(In reply to youenn fablet from comment #9)
> > crop the video after "flipping" 3-5 times. However, once the video is
> 
> Doing your test, it seems to work for me after clicking 2 times.

I'm seeing the following in portrait:

Click 1  - No change
Click 2  - Crops to 4:3
Click 3  - No change
Click 4  - No change
Click 5+ - Always crops, when in 3:4 aspect ratio the video is "zoomed" compared to the original 3:4 video.