Bug 214078

Summary: REGRESSION: (r263222-263223): [ macOS wk2 debug ] accessibility/textarea-selected-text-range.html is flaky failing
Product: WebKit Reporter: Karl Rackler <rackler>
Component: New BugsAssignee: Geoffrey Garen <ggaren>
Status: RESOLVED FIXED    
Severity: Normal CC: aboxhall, andresg_22, apinheiro, cfleizach, darin, dmazzoni, ews-watchlist, ggaren, jacob_uphoff, jcraig, jdiggs, Lawrence.j, rackler, samuel_white, webkit-bot-watchers-bugzilla, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
Patch
none
Patch none

Description Karl Rackler 2020-07-08 05:57:46 PDT 
https://results.webkit.org/?suite=layout-tests&test=accessibility%2Ftextarea-selected-text-range.html&limit=50000&platform=mac&flavor=wk2&style=debug

Description:
The test is flaky failing on [ macOS wk2 debug ] release.  It was passing consistently until 06/19/2020.

History: 
https://results.webkit.org/?suite=layout-tests&test=accessibility%2Ftextarea-selected-text-range.html&limit=50000&platform=mac&flavor=wk2&style=debug
Diff: 
--- /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/accessibility/textarea-selected-text-range-expected.txt
+++ /Volumes/Data/slave/catalina-debug-tests-wk2/build/layout-test-results/accessibility/textarea-selected-text-range-actual.txt
@@ -1,5 +1,5 @@
 {4, 0}
-{8, 2}
+{4, 0}
 {25, 0}
 PASS successfullyParsed is true
Comment 1 Radar WebKit Bug Importer 2020-07-08 05:58:16 PDT 
<rdar://problem/65220756>
Comment 2 Karl Rackler 2020-07-08 06:11:42 PDT 
Created attachment 403774 [details]
Patch
Comment 3 Karl Rackler 2020-07-08 06:17:31 PDT 
I have marked this is a flaky test while the issue is investigated.
Comment 4 EWS 2020-07-08 09:22:52 PDT 
Tools/Scripts/svn-apply failed to apply attachment 403774 [details] to trunk.
Please resolve the conflicts and upload a new patch.
Comment 5 Karl Rackler 2020-07-08 10:48:29 PDT 
I can reproduce this on r263223, but unable to reproduce on r263222.

Commit: r264039 

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result:
[32/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[36/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[37/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 29 tests run.


26 tests ran as expected, 3 didn't (471 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]


Commit: r263224

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
[47/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[52/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[53/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 42 tests run.


39 tests ran as expected, 3 didn't (458 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]

Commit: r223222

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
All 500 tests ran as expected.
Comment 6 Karl Rackler 2020-07-08 11:05:14 PDT 
Start on Geoff because the issue looks like it may be related to the  "[Cocoa] Build callOnMainThread on WTF::RunLoop rather than on
a timer" in r263223

Commit: r263223

Command: run-webkit-tests --debug --iterations 500 --exit-after-n-failures 3 --debug --force -f accessibility/textarea-selected-text-range.html

Result: 
[47/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[52/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
[53/500] accessibility/textarea-selected-text-range.html failed unexpectedly (text diff)
Exiting early after 3 failures. 42 tests run.


39 tests ran as expected, 3 didn't (458 didn't run):


Regressions: Unexpected text-only failures (1)
  accessibility/textarea-selected-text-range.html [ Failure ]
Comment 7 Jason Lawrence 2020-07-08 11:45:54 PDT 
I have marked this test as flaky failing while this issue is investigated.
https://trac.webkit.org/changeset/264123/webkit
Comment 8 Geoffrey Garen 2020-07-08 12:30:08 PDT 
Created attachment 403802 [details]
Patch
Comment 9 Darin Adler 2020-07-08 12:33:31 PDT 
Comment on attachment 403802 [details]
Patch

View in context: https://bugs.webkit.org/attachment.cgi?id=403802&action=review

> LayoutTests/ChangeLog:8
> +        Make this test deterministic by waiting for the correct result.

With no time limit other than the test runner’s own limit!
Comment 10 EWS 2020-07-08 12:54:07 PDT 
Committed r264129: <https://trac.webkit.org/changeset/264129>

All reviewed patches have been landed. Closing bug and clearing flags on attachment 403802 [details].