Bug 218891

Summary: element.rect doesn't yield correct co-ordinates on iOS
Product: WebKit Reporter: Luke Hill <lukehill_uk>
Component: WebDriverAssignee: Nikita Vasilyev <nvasilyev>
Status: NEW ---    
Severity: Normal CC: bburg, darin, simon.fraser, smoley, webkit-bug-importer, zalan
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
[WIP] Patch v1.0 bburg: review?

Description Luke Hill 2020-11-13 01:31:03 PST 
I couldn't find the right location now to submit these.

After initially submitting on selenium, then on appium, I'm now here. `element.rect` does not yield correct results on iOS compared to other browsers

For full repro see: https://github.com/SeleniumHQ/selenium/issues/8869 / https://github.com/appium/ruby_lib_core/issues/285

```
Bad candidates

iPhone 8 - iOS 12 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=293, height=48>
iPhone 8 - iOS 13 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=293, height=48>
iPhone 11 - iOS 13 - #<struct Selenium::WebDriver::Rectangle x=17, **y=1**, width=332, height=48>
Good candidates

Windows 7 - Chrome v83 - #<struct Selenium::WebDriver::Rectangle x=17, y=**1032**, width=750, height=48>
Windows 10 - Chrome v86 - #<struct Selenium::WebDriver::Rectangle x=17, y=**1060**, width=750, height=48>
Windows 10 - Firefox v80 - #<struct Selenium::WebDriver::Rectangle x=17.0, y=**1010.0**, width=750.0, height=48.0>
OSX Mojave - Safari 12 - #<struct Selenium::WebDriver::Rectangle x=17, y=**820**, width=750, height=48>
```
Comment 1 Luke Hill 2020-11-16 01:50:47 PST 
NB: During some parts of the testing. It's even possible to get the y ordinate to be negative in iOS. So something is definitely going wrong here!!
Comment 2 Radar WebKit Bug Importer 2020-11-16 18:23:48 PST 
<rdar://problem/71468283>
Comment 3 BJ Burg 2020-11-16 22:18:41 PST 
Hello Luke, thanks for writing.

It's not quite clear to me how you are scrolling. Could you please attach a test case so that there is no ambiguity?

In general, iOS coordinates are computed and returned in visual viewport space, but the Get Element Rect command explicitly wants to add the scrollY to this coordinate system. In this case the scrollY of ~1000 is missing, so this indeed looks like a driver bug.
Comment 4 BJ Burg 2020-11-16 22:23:18 PST 
From reading the code, it looks like we need to change WebAutomationSessionProxy::computeElementLayout() for the CoordinateSystem::Page case, so that it shifts the element bounds by window.scrollX and window.scrollY.

Spec language is here: https://w3c.github.io/webdriver/#dfn-calculate-the-absolute-position
Comment 5 Luke Hill 2020-11-17 04:08:23 PST 
I do use a JS "hack" to scroll. Which perhaps isn't computing well with the iOS Driver.

But it works (i.e. it does scroll). For ref I do a window.scrollTo(10000,0) call.

I "know" for iOS I could maybe do something different, but we don't care how the user scrolls - As for that webpage, we want to test it "stickies" the OISC Warning. To me this was the "best" way to test the sticky behaviour.

I.e. the rect should be moved from its starting position. So my test grabs the starting y value, scrolls down, asserts it has changed. But for iOS it goes all weird. It works fine on Mojave 12 Safari so it suggests it's not globally broken with the safaridriver. So I'm not sure.
Comment 6 BJ Burg 2021-04-23 16:19:38 PDT 
Created attachment 426962 [details]
[WIP] Patch v1.0
Comment 7 Darin Adler 2021-04-25 10:47:06 PDT 
Comment on attachment 426962 [details]
[WIP] Patch v1.0

View in context: https://bugs.webkit.org/attachment.cgi?id=426962&action=review

> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:662
> +        // XXX

XXX?
Comment 8 Simon Fraser (smfr) 2021-04-26 09:22:30 PDT 
Comment on attachment 426962 [details]
[WIP] Patch v1.0

View in context: https://bugs.webkit.org/attachment.cgi?id=426962&action=review

> Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp:668
> +        scrollOffset.scale(2); // convertRectFromFrameClientToRootView subtracts scrollX/Y and we need to add it back and then again so it's positive

This is very weird.