Bug 242128

Summary: REGRESSION (251741@main-251739@main?): [ iOS ] TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing is a flaky failure
Product: WebKit Reporter: Karl Rackler <rackler>
Component: New BugsAssignee: Nobody <webkit-unassigned>
Status: NEW ---    
Severity: Normal CC: ap, honamluxurychef, ryanhaddad, webkit-bot-watchers-bugzilla, webkit-bug-importer, wenson_hsieh
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
See Also: https://bugs.webkit.org/show_bug.cgi?id=241890

Description Karl Rackler 2022-06-29 12:53:38 PDT 
Description:
TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing

The first failure that I saw on the dashboard was on 6/22/2022 at 251749@main.

REPRODUCTION STEPS
I can reproduce this on 251741@main, but unable to reproduce it on 251738@main or earlier.  This affects the iOS model iPhone 8 only. 

Command: 
run-api-tests --release --verbose --iterations 10 --model iPhone8 TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing

Result: 
Same as bot below

History:
https://results.webkit.org/?suite=api-tests&test=TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing&limit=50000&platform=ios&model=iPhone%208

run-api-tests stdio:
Failed
    TestWebKitAPI.AutocorrectionTests.AutocorrectionContextBeforeAndAfterEditing
        
        /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-Build/build/Tools/TestWebKitAPI/Tests/ios/AutocorrectionTestsIOS.mm:182
        Expected equality of these values:
          "hello"
          [contextAfterInsertingText contextBeforeSelection]
            Which is: ""
        
        
        /Volumes/Data/worker/Apple-iOS-15-Simulator-Release-Build/build/Tools/TestWebKitAPI/Tests/ios/AutocorrectionTestsIOS.mm:187
        Expected equality of these values:
          "hello"
          [contextAfterSelecting selectedText]
            Which is: ""
Comment 1 Radar WebKit Bug Importer 2022-06-29 12:53:56 PDT 
<rdar://problem/96161911>
Comment 2 Karl Rackler 2022-06-29 13:03:04 PDT 
I have disabled this test while this issue is investigated.
Comment 3 EWS 2022-06-29 13:11:29 PDT 
Test gardening commit 251961@main (9aa946bf2540): <https://commits.webkit.org/251961@main>

Reviewed commits have been landed. Closing PR #1912 and removing active labels.
Comment 4 Karl Rackler 2022-06-30 10:28:07 PDT 
The test failure is referencing contextAfterInsertingText and contextAfterSelecting from 246659@main.  

The flakiness dashboard shows a clear change from consistently passing to flaky failure first appearing on the dashboard at 251749@main.  

Reproduction steps further reduced the range(251741@main-251739@main), but it is unclear what in this range may have caused this.
Comment 5 Wenson Hsieh 2022-06-30 12:38:19 PDT 
(In reply to Karl Rackler from comment #4)
> The test failure is referencing contextAfterInsertingText and
> contextAfterSelecting from 246659@main.  
> 
> The flakiness dashboard shows a clear change from consistently passing to
> flaky failure first appearing on the dashboard at 251749@main.  
> 
> Reproduction steps further reduced the range(251741@main-251739@main), but
> it is unclear what in this range may have caused this.

Yeah, neither of the commits in that range remotely seem like they could've caused these test failures.

Were there any interesting configuration changes might've affected the bots at around the same time? (i.e. upgrading to a different version, or using new simulator device models, etc.)
Comment 6 Ryan Haddad 2022-07-01 12:58:41 PDT 
(In reply to Wenson Hsieh from comment #5)
> Were there any interesting configuration changes might've affected the bots
> at around the same time? (i.e. upgrading to a different version, or using
> new simulator device models, etc.)

We have not made any changes in this area recently, we're also seeing this failure on internal bots in roughly the same regression range http://commits.webkit.org/compare/251744@main...251736@main
Comment 7 Ryan Haddad 2022-07-01 12:59:52 PDT 
Not sure what happened with my URL in comment 6, it should be showing a range of 251744@main - 251736@main
http://commits.webkit.org/compare/251744@main...251736@main