Bug 248533

We are having lots of deadlocks in the test runs recently.

1) Check the EWS queue:

https://ews-build.webkit.org/#/builders/35

Many of them are ending with "Unexpected infrastructure issue: The layout-test run with change generated no list of results and exited with error, retrying with the hope it was a random infrastructure error."
that is caused because the layout test steps deadlocks and the timeout of the step triggers

2) Same for the post-commit bots. Example WPE release tests:

https://build.webkit.org/#/builders/40

The layout test is not finishing, deadlocks


I have been debugging this and if you run the tests with only one worker (NUMBER_OF_PROCESSORS=1) instead of the deadlock you get this backtrace:


BrokenPipeError raised: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 186, in write
    self._proc.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 92, in main
    run_details = run(port, options, args, stderr)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/run_webkit_tests.py", line 508, in run
    run_details = manager.run(args)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 413, in run
    temp_initial_results, temp_retry_results, temp_enabled_pixel_tests_in_retry = self._run_test_subset(test_inputs, device_type=device_type)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 487, in _run_test_subset
    initial_results = self._run_tests(test_inputs, self._options.repeat_each, self._options.iterations, int(self._options.child_processes), retrying=False, device_type=device_type)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py", line 570, in _run_tests
    return self._runner.run_tests(self._expectations[device_type], new_test_inputs, num_workers, retrying, device_type)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 176, in run_tests
    pool.do(
  File "/app/webkit/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 405, in do
    result = function(*args, **kwargs)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 77, in run_shard
    return Worker.instance.run_tests(shard)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 319, in run_tests
    Worker.instance.run_test(input, shard.name)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 347, in run_test
    result = self._run_test_with_or_without_timeout(test_input, test_timeout_sec, stop_when_done)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 404, in _run_test_with_or_without_timeout
    return self._run_test_in_this_thread(test_input, stop_when_done)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 490, in _run_test_in_this_thread
    return self._run_single_test(self._driver, test_input, stop_when_done)
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/layout_test_runner.py", line 493, in _run_single_test
    return single_test_runner.run_single_test(
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 48, in run_single_test
    return runner.run()
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 128, in run
    return self._run_compare_test()
  File "/app/webkit/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py", line 131, in _run_compare_test
    driver_output = self._driver.run_test(self._driver_input(), self._stop_when_done)
  File "/app/webkit/Tools/Scripts/webkitpy/port/driver.py", line 863, in run_test
    return self._driver.run_test(driver_input, stop_when_done)
  File "/app/webkit/Tools/Scripts/webkitpy/port/driver.py", line 233, in run_test
    self._server_process.write(command)
  File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 188, in write
    self.stop(0.0)
  File "/app/webkit/Tools/Scripts/webkitpy/port/server_process.py", line 403, in stop
    self._proc.stdin.close()
BrokenPipeError: [Errno 32] Broken pipe



The issue seems to be we are not handling correctly the BrokenPipeError() when a test crashes.

Not sure what change lead to this but maybe it was a Python upgrade in the Flatpak SDK
We are running Python 3.10.6 right now