Bug 239780

Summary:	A python error causes results.html to be missing from layout test results
Product:	WebKit	Reporter:	Simon Fraser (smfr) <simon.fraser>
Component:	Tools / Tests	Assignee:	Jonathan Bedard <jbedard>
Status:	NEW ---
Severity:	Normal	CC:	aakash_jain, ap, jbedard, simon.fraser, webkit-bug-importer
Priority:	P2	Keywords:	InRadar
Version:	Safari Technology Preview
Hardware:	Unspecified
OS:	Unspecified
See Also:	https://bugs.webkit.org/show_bug.cgi?id=255873

Description Simon Fraser (smfr) 2022-04-26 11:40:12 PDT

This EWS run <https://ews-build.webkit.org/#/builders/68/builds/13819/steps/17/logs/stdio> shows an error shutting down the simulator:

10:08:27.729 681 Removing device 'Managed 7'
10:08:27.897 681 "/usr/bin/xcrun simctl delete 3882575F-7118-4051-9D0F-105AEEC707AE" took 0.17s
TypeError raised: 'NoneType' object is not subscriptable
Traceback (most recent call last):
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 495, in run
    run_details = manager.run(args)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 506, in _run_test_subset
    retry_results = self._run_tests(retry_test_inputs,
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 566, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 175, in run_tests
    pool.do(
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 400, in do
    result = function(*args, **kwargs)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 76, in run_shard
    return Worker.instance.run_tests(shard)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 318, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 346, in run_test
    result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 403, in _run_test_with_or_without_timeout
    return self._run_test_in_this_thread(test_input, stop_when_done)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 489, in _run_test_in_this_thread
    return self._run_single_test(self._driver, test_input, stop_when_done)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 492, in _run_single_test
    return single_test_runner.run_single_test(
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test
    return runner.run()
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 123, in run
    return self._run_reftest()
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 340, in _run_reftest
    test_result = self._compare_output_with_reference(reference_output, test_output, reference_filename, expectation == '!=')
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 446, in _compare_output_with_reference
    diff_result.passed = self._test_passes_fuzzy_matching(fuzzy_tolerance, diff_result.fuzzy_data)
  File "/Volumes/Data/worker/iOS-15-Simulator-WK2-Tests-EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching
    actualMaxDifference = fuzzy_result['max_difference']
TypeError: 'NoneType' object is not subscriptable
10:08:28.673 681 Stopping HTTP server ...
10:08:28.929 681 Stopping WebSocket server ...
10:08:29.139 681 Stopping Web Platform Test server ...
program finished with exit code 254

and this means that the script fails to generate results.html, so the entire run is wasted.

Comment 1 Radar WebKit Bug Importer 2022-05-03 11:41:15 PDT

<rdar://problem/92683094>

Comment 2 Jonathan Bedard 2022-05-03 13:28:49 PDT

That doesn't look like a failure shutting down simulators, that's a failure in one of our child processes in Python code when retrying, this bit is probably the key:

EWS/build/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 401, in _test_passes_fuzzy_matching
    actualMaxDifference = fuzzy_result['max_difference']
TypeError: 'NoneType' object is not subscriptable

Comment 3 Simon Fraser (smfr) 2022-05-03 13:39:59 PDT

Ah I missed that. I wonder why this just happens sometimes.