Bug 243989

Summary: Tools/Scripts/test-webkitpy-python2 is flaky (Cannot send message to closed queue)
Product: WebKit Reporter: Kimmo Kinnunen <kkinnunen>
Component: Tools / TestsAssignee: Jonathan Bedard <jbedard>
Status: NEW ---    
Severity: Normal CC: aakash_jain, ap, gsnedders, jbedard, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: Other   
Hardware: Unspecified   
OS: Unspecified   

Description Kimmo Kinnunen 2022-08-16 05:33:35 PDT 
Tools/Scripts/test-webkitpy-python2 is flaky (Cannot send message to closed queue)

Blocked my commit:
https://ews-build.webkit.org/#/builders/5/builds/89742

Local runs produce flaky results:
1. success
2. repro of the Cannot send message to closed queue
3. exception but success


kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2 
Suppressing most webkitpy logging while running unit tests.
Deleting orphan *.pyc file: /Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/xcode/__pycache__/__init__.cpython-39.pyc
....
Deleting orphan *.pyc file: /Users/kkinnunen/WebKit/OpenSource/Source/WebKit/Scripts/webkit/__pycache__/parser.cpython-39.pyc
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[752/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_reftest_driver_should_run_expected_mismatch_html passedProcess Process-8:4:
Traceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main
    cls.queue.close()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close
    self.outgoing.join_thread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread
    self._jointhread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join
    thread.join()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler
    with StackTraceFileContext(output_file=output_file) as file:
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__
    self.file = open(self.file_name, 'w')
IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/41087-python_stack_trace.txt'
[789/2279] webkitpy.layout_tests.views.metered_stream_unittest.RegularTest.test_log_after_update passedworker/0 starting              
worker/1 starting
worker/0 stopping
worker/1 stopping
[810/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+8)Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.825s                                                                                                                                

OK
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[748/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_reftest_expected_html_should_be_ignored passedUnable to flush stdout: Broken pipe
Unable to flush stdout: Broken pipe
Unable to flush stdout: Broken pipe
Unable to flush stdout: Broken pipe
Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Unable to flush stdout: Broken pipe
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Cannot send message to closed queue
Traceback (most recent call last):
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 366, in <module>
    sys.exit(main())
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 115, in main
    return not tester.run()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 188, in run
    return self._run_tests(names)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/main.py", line 228, in _run_tests
    test_runner.run(parallel_tests, getattr(self._options, 'child_processes', 1))
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/runner.py", line 53, in run
    pool.run(('test', test_name) for test_name in test_names)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 104, in run
    self.wait()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 139, in wait
    self._loop(block=True)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/message_pool.py", line 199, in _loop
    self._caller.handle(message.name, message.src, *message.args)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/runner.py", line 71, in handle
    self.printer.print_finished_test(source, test_name, delay, failures, errors)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/test/printer.py", line 149, in print_finished_test
    self.running_tests.remove(test_name)
ValueError: list.remove(x): x not in list
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[783/2279] webkitpy.performance_tests.perftest_unittest.TestPerfTest.test_parse_output_with_description passedworker/1 starting                   
worker/0 starting
worker/1 stopping
worker/0 stopping
[817/2279] webkitpy.performance_tests.perftest_unittest.TestSingleProcessPerfTest.test_use_only_one_process passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.181s                                                                                                                              

OK
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[732/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_ews_corner_case_skipped_test passedProcess Process-7:7:           
Traceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main
    cls.queue.close()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close
    self.outgoing.join_thread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread
    self._jointhread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join
    thread.join()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler
    with StackTraceFileContext(output_file=output_file) as file:
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__
    self.file = open(self.file_name, 'w')
IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/43178-python_stack_trace.txt'
[756/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RebaselineTest.test_missing_results passedworker/1 starting
worker/0 starting
worker/1 stopping
worker/0 stopping
[802/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+7)Unknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.286s                                                                                                                              

OK
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[742/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_keyboard_interrupt passedworker/1 starting                          
worker/0 starting
worker/0 stopping
worker/1 stopping
[827/2279] webkitpy.performance_tests.perftestsrunner_unittest.MainTest.test_parse_args passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.270s                                                                                                                        

OK
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[727/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_ews_corner_case_failing_test passedProcess Process-5:6:         
Traceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main
    cls.queue.close()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close
    self.outgoing.join_thread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread
    self._jointhread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join
    thread.join()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler
    with StackTraceFileContext(output_file=output_file) as file:
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__
    self.file = open(self.file_name, 'w')
IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/44593-python_stack_trace.txt'
[777/2279] webkitpy.common.system.executive_unittest.ExecutiveTest.test_run_command_with_unicode (+6)worker/0 starting 
worker/1 starting
worker/0 stopping
worker/1 stopping
[836/2279] webkitpy.performance_tests.perftestsrunner_integrationtest.MainTest.test_run_crash_test passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.277s                                                                                                                              

OK
kkinnunen@mpb4-local OpenSource % Tools/Scripts/test-webkitpy-python2
Suppressing most webkitpy logging while running unit tests.
Skipping tests in the following modules or packages because they are really, really, slow:
    webkitpy.common.checkout.scm.scm_unittest
    (https://bugs.webkit.org/show_bug.cgi?id=31818; use --all to include)

[722/2279] webkitpy.layout_tests.models.test_run_results_unittest.SummarizedResultsTest.test_summarized_results_world_leaks_disabled passedProcess Process-5:7:
Traceback (most recent call last):
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 313, in main
    cls.queue.close()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/libraries/webkitcorepy/webkitcorepy/task_pool.py", line 161, in close
    self.outgoing.join_thread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 171, in join_thread
    self._jointhread()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/util.py", line 207, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 218, in _finalize_join
    thread.join()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/AppleInternal/Library/Frameworks/PythonLegacy.framework/Versions/2.7/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 74, in handler
    with StackTraceFileContext(output_file=output_file) as file:
  File "/Users/kkinnunen/WebKit/OpenSource/Tools/Scripts/webkitpy/common/interrupt_debugging.py", line 54, in __enter__
    self.file = open(self.file_name, 'w')
IOError: [Errno 2] No such file or directory: '/tmp/layout-test-results/45397-python_stack_trace.txt'
[783/2279] webkitpy.layout_tests.run_webkit_tests_integrationtest.RunTest.test_random_order_test_specified_multiple_times passedworker/1 starting
worker/0 starting
worker/1 stopping
worker/0 stopping
[842/2279] webkitpy.performance_tests.perftestsrunner_integrationtest.MainTest.test_run_test_set_for_parser_tests passedUnknown browser mosaic. Defaulting to Cog and MiniBrowser selection
Ran 2279 tests in 8.423s                                                                                                                   

OK
kkinnunen@mpb4-local OpenSource %
Comment 1 Radar WebKit Bug Importer 2022-08-23 05:34:15 PDT 
<rdar://problem/99028920>
Comment 2 Aakash Jain 2022-08-23 08:05:37 PDT 
> Blocked my commit
Note that webkitpy queue doesn't block PRs. Some other queue (e.g.: builder or layout-test queue) would have blocked the PR.
Comment 3 Sam Sneddon [:gsnedders] 2024-03-22 07:47:09 PDT 
My guess is that one of the subprocesses is crashing (and older multiprocessing doesn't handle workers going away unexpectedly), causing this.

I do wonder if we should just move to running tests with -j1 on Python 2, like we already do on Windows.

Locally this is a decent perf regression, but maybe worth it for the stability:

```
% hyperfine -r1 -L children 1,8 './Tools/Scripts/test-webkitpy-python2 -j{children}' 
Benchmark 1: ./Tools/Scripts/test-webkitpy-python2 -j1
  Time (abs ≡):        89.726 s               [User: 41.462 s, System: 15.612 s]
 
Benchmark 2: ./Tools/Scripts/test-webkitpy-python2 -j8
  Time (abs ≡):        25.100 s               [User: 43.881 s, System: 20.178 s]
 
Summary
  ./Tools/Scripts/test-webkitpy-python2 -j8 ran
    3.57 times faster than ./Tools/Scripts/test-webkitpy-python2 -j1
```

This is especially true when the majority of PRs don't run the webkitpy tests at all, so the bot is largely free.

(Note however my local machine is much faster than what we're running in EWS: the -j8 time is nearly twice as fast, but a 4x multiplier would take us from 1 minute on EWS to run the Python 2 tests to 4 minutes, which is probably still acceptable.)