Bug 206932

Summary: run-safari / run-webkit-tests --ios-simulator not working with XCode 13.3.1 (11C504)
Product: WebKit Reporter: Frédéric Wang (:fredw) <fred.wang>
Component: Tools / TestsAssignee: Jonathan Bedard <jbedard>
Status: RESOLVED FIXED    
Severity: Normal CC: ajuma, ap, cathiechen, ews-watchlist, glenn, jbedard, jlewis3, ryanhaddad, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Output of xcrun simctl list
none
Patch
none
Patch none

Description Frédéric Wang (:fredw) 2020-01-29 07:41:13 PST
run-safari hangs without launching simulator, I'm not sure why.

For run-webkit-tests, traceback is shown below. In get_runtime_for_device_type() the device_type is software_variant="iOS" and software_version="13.2" but SimulatedDeviceManager.AVAILABLE_RUNTIMES only has "13.3" versions. A ugly workaround is to set device_type.software_version = None, which allows to run the tests for me.

Not sure that really helps, but I remember that in previous XCode version run-safari was actually opening two versions of the simulator which seemed a bug.

AssertionError raised: 
Traceback (most recent call last):
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 91, in main
    run_details = run(port, options, args, stderr)
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 485, in run
    run_details = manager.run(args)
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 270, in run
    if not self._set_up_run(tests_to_run_by_device[device_type], device_type=device_type):
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 176, in _set_up_run
    self._port.setup_test_run(device_type)
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/port/device_port.py", line 179, in setup_test_run
    reboot=self.get_option('reboot', False),
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 404, in initialize_devices
    device = SimulatedDeviceManager._create_or_find_device_for_request(request, host, name_base)
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 264, in _create_or_find_device_for_request
    device_type = SimulatedDeviceManager._disambiguate_device_type(request.device_type)
  File "/Users/fred/WebKit/Tools/Scripts/webkitpy/xcode/simulated_device.py", line 226, in _disambiguate_device_type
    assert runtime is not None
AssertionError
Comment 1 Jonathan Bedard 2020-01-29 08:33:12 PST
What's the output of 'xcrun simctl list'?
Comment 2 Frédéric Wang (:fredw) 2020-01-29 08:50:54 PST
Created attachment 389142 [details]
Output of xcrun simctl list

Here is what I get.

Maybe it's also worth noting that I don't see "iPhone SE for WebKit development" (or whathever how it is called) anymore when using the simulator UI.
Comment 3 Jonathan Bedard 2020-01-29 09:01:29 PST
(In reply to Frédéric Wang (:fredw) from comment #2)
> Created attachment 389142 [details]
> Output of xcrun simctl list
> 
> Here is what I get.
> 
> Maybe it's also worth noting that I don't see "iPhone SE for WebKit
> development" (or whathever how it is called) anymore when using the
> simulator UI.

We have a bot exhibiting this too. Try running 'xcrun simctl delete unavailable', you have a bunch of simulators for old SDK versions. I'm going to try the same thing on our bot.
Comment 4 Jonathan Bedard 2020-01-29 09:11:35 PST
(In reply to Jonathan Bedard from comment #3)
> (In reply to Frédéric Wang (:fredw) from comment #2)
> > Created attachment 389142 [details]
> > Output of xcrun simctl list
> > 
> > Here is what I get.
> > 
> > Maybe it's also worth noting that I don't see "iPhone SE for WebKit
> > development" (or whathever how it is called) anymore when using the
> > simulator UI.
> 
> We have a bot exhibiting this too. Try running 'xcrun simctl delete
> unavailable', you have a bunch of simulators for old SDK versions. I'm going
> to try the same thing on our bot.

Not going to be that simple, this will need a code change, hopefully something minor.
Comment 5 Jonathan Bedard 2020-01-29 09:46:44 PST
This is probably an issue with Xcode, since what causes this is having the iOS 13.2 SDK installed but the iOS 13.3 simulator runtime. For a work-around, Xcode -> Preferences -> Components, install iOS 13.2 simulator runtime. Things should work.
Comment 6 Jonathan Bedard 2020-01-29 15:14:57 PST
Created attachment 389193 [details]
Patch
Comment 7 Jonathan Bedard 2020-01-29 15:19:06 PST
(In reply to Jonathan Bedard from comment #6)
> Created attachment 389193 [details]
> Patch

Had a discussion with Alexey where he pointed out that runtime versions and SDK versions don't need to remain in lockstep. This is actually just a more generalized case of what we were already doing with allowing a request for just a major version to match the minor versions too.
Comment 8 Alexey Proskuryakov 2020-01-29 15:25:29 PST
Comment on attachment 389193 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=389193&action=review

> Tools/Scripts/webkitpy/xcode/simulated_device.py:206
> +        # Return the available runtime closest to specified runtime

I don't think that "closest" explains it well. We should either call it "best match", or actually explain what the code is trying to achieve. Also, please add a period at the end of the sentence.
Comment 9 Jonathan Bedard 2020-01-29 15:47:32 PST
Created attachment 389195 [details]
Patch
Comment 10 Jonathan Bedard 2020-01-29 16:59:08 PST
Committed r255404: <https://trac.webkit.org/changeset/255404>
Comment 11 Radar WebKit Bug Importer 2020-01-29 17:00:18 PST
<rdar://problem/59013211>